Terraform 0.14: The Dependency Lock File

Terraform
1

Ever since Terraform has had support for installing external dependencies (first external modules in early Terraform, and then separately-released providers in Terraform 0.10) it has used a hidden directory under .terraform as a sort of local, directory-specific “lock” of the selected versions of those dependencies. If you ran terraform init again in the same working directory then Terraform would select those same versions again.

Unfortunately this strategy hasn’t been sufficient for today’s most common usage patterns for Terraform: many people on a team all using Terraform together, either on their separate workstations (so each has their own working directory) or in an ephemeral remote execution environment (so an entirely new working directory is created for each run).

Terraform 0.14 aims to address this by introducing a new model for tracking dependency selections: a new file in your root module which you can include in your version control system so that you can communicate about changes to dependency versions in the same way as you might communicate about changes to your main configuration: through a code review process.

The first iteration of this for Terraform 0.14 covers only provider dependencies. Tracking version selections for external modules will hopefully follow in a later release, but requires some deeper design due to Terraform’s support for a large number of different installation methods for external modules.

When you run terraform init for the first time after upgrading to Terraform 0.14, Terraform will generate a new file .terraform.lock.hcl in the current working directory, alongside the existing local cache directory, .terraform. This new dependency lock file is intended to retain metadata about the external items that Terraform will fetch into the .terraform directory, so that you can include the lock file in your version control while continuing to exclude the .terraform directory.

Once the lock file is generated, terraform init can make two additional guarantees on future runs:

  • It will always select the same version of each provider, even if a newer release becomes available that would also be accepted by the configured version constraints.

    This separates the idea of which provider versions a module/configuration is compatible with (the version constraints) from which versions a configuration is currently using (the lock file).

  • It will check the downloaded provider packages against the checksums it saw the first time you saw the provider, raising an error if any package does not match.

    This therefore implements a “trust on first use” model where you can, if you wish, perform audits or other checks on new provider versions you intend to use and then use the lock file to remember the checksums of the packages you audited, so you’ll know if the upstream provider package is modified in some way after you reviewed it.

You can run terraform init -upgrade to turn off the new rules described above and have Terraform instead select the newest available version of each provider that matches the version constraints. In that case Terraform will still update the dependency lock file to match the new selections, so you can use your version control system to review the changes to that file and decide if you want to keep those new selections.

As much as possible we’ve aimed to keep the new behavior consistent with the old behavior if we imagine the contents of the .terraform directory as formerly being a sort of implied “lock file” that survived only as long as the working directory it was created in.

The new dependency lock file formalizes that behavior by recording the same information in a file you can easily keep under version control, and so the lock file is now the authority for what is selected for a particular configuration, and the .terraform directory is in turn just a local cache of remote the items the lock file describes.

Although the dependency lock file is primarily an artifact maintained automatically by Terraform itself, we’ve used the familiar HCL syntax for it because we expect teams to see diffs of this file during their code review processes and we want those diffs to be easy to read and understand. The following is an example dependency lock file recording a single provider dependency:

provider "registry.terraform.io/hashicorp/azurerm" {
  version     = "2.1.0"
  constraints = "~> 2.1.0"
  hashes = [
    "h1:EOJImaEaVThWasdqnJjfYc6/P8N/MRAq1J7avx5ZbV4=",
    "zh:0015b491cf9151235e57e35ea6b89381098e61bd923f56dffc86026d58748880",
    "zh:4c5682ba1e0fc7e2e602d3f103af1638f868c31fe80cc1a884a97f6dad6e1c11",
    "zh:57bac885b108c91ade4a41590062309c832c9ab6bf6a68046161636fcaef1499",
    "zh:5810d48f574c0e363c969b3f45276369c8f0a35b34d6202fdfceb7b85b3ac597",
    "zh:5c6e37a44462b8662cf9bdd29ce30523712a45c27c5d4711738705be0785db41",
    "zh:64548940a3387aa3a752e709ee9eb9982fa820fe60eb60e5f212cc1d2c58549e",
    "zh:7f46749163da17330bbb5293dc825333c86304baa0a7c6256650ac536b4567c8",
    "zh:8f8970f2df75ac43ffdd112055ee069d8bd1030f7eb4367cc4cf494a1fa802c3",
    "zh:9ad693d00dc5d7d455d06faba70e716bce727c6706f7293288e87fd7956b8fe0",
    "zh:b6e3cb55e6aec62b47edd0d2bd5e14bd6a2bcfdac65930a6e9e819934734c57b",
    "zh:d6a3f3b9b05c28ecf3919e9e7afa185805a6d7442fc4b3eedba749c2731d1f0e",
    "zh:d81fb624a357c57c7ea457ce543d865b39b12f26c2edd58a2f7cd43326c91010",
  ]
}

The checksum verification feature works best if you do your initial installation from a provider’s origin registry, because Terraform can then record all of the checksums that were cryptographically signed by the original author. However, checksum verification is also supported for installation from filesystem and network mirrors, and Terraform 0.14 includes a new command terraform providers lock that may help you seed a lock file with official upstream checksums in preparation for later installation from your own mirror.

The dependency lock file mechanism will be included in all of the beta releases of Terraform 0.14, and we’d appreciate your early feedback so we can address any rough edges prior to the final 0.14.0 release.

Because the dependency lock behavior is centered primarily around the provider installation step in the terraform init command, you can test it against your existing configurations safely, as long as you don’t change your backend configuration and you don’t subsequently run any other commands that could modify your state. The terraform validate command is a good choice to run after terraform init to verify that Terraform is able to find and communicate with the installed plugins while avoiding any external side-effects.

After you have tested, we recommend that you discard the generated .terraform.lock.hcl file, rather than including it in your version control main branch, because feedback during the beta process may cause us to change the format of the lock file slightly before final release. Once the lock file format has been included in a stable release, new Terraform releases will remain compatible with it.

You may also need to delete the .terraform cache directory after you conclude your beta testing, because the file structure created in there by Terraform 0.14 may not be forward-compatible with all previous versions of Terraform. You can run terraform init again with your previous Terraform version to create a new .terraform cache directory.

We have some ideas for things to test listed below, but it’s also very helpful to try some situations we didn’t consider, in case there are opportunities to improve the behavior for less common situations.

  • Is terraform init able to correctly determine all of the providers your configuration depends on and record its version selections in the lock file? (Note: The lock file intentionally excludes the terraform.io/builtin/terraform provider because it’s always distributed as part of Terraform CLI itself, not as a separately-installed plugin.)
  • If you use either a local filesystem mirror or a network mirror to distribute providers to your Terraform execution environments, is terraform init still able to install from those mirrors and verify the package checksums against them?
    • Furthermore, are you able to use the terraform providers lock command to populate the lock file with the official checksums from each provider’s origin registry, and then have terraform init successfully verify your mirrored packages against those checksums?
  • If you change a version constraint in your configuration to permit a newer version (or remove the constraint entirely) after the lock file is populated, does terraform init with no arguments still install the previously-selected version? Can you then run terraform init -upgrade and see Terraform select the newer available version instead?
  • If you are in an environment where people run Terraform on different platforms, such as some folks using Windows and other folks using macOS, can you commit your dependency lock file to a temporary branch in your repository and then have a colleague who uses a different platform check out that branch and run terraform init? Does Terraform still successfully follow the selections recorded in the lock file, even though the second person needs to use a different build of the provider plugin?

If you have feedback to share, please feel free to reply to this topic! If some feedback spurs a longer discussion than I may split that discussion out into a separate topic so we can keep this one easier to follow, but starting here will make it easier to find all of the relevant feedback in one place.

Thanks!

3 Likes
4

3 posts were split to a new topic: Development overrides for providers under development

5

Hi there,

I am in the last situation: terraform on multiple OS/arch.

I am not sure I am doing the right thing but the usual workflow (terraform init, plan, apply; git add,commit, push) is broken.

Here is what’s happening:

  • Darwin/amd64: terraform init, plan, apply, git add,commit,push
  • Linux/arm64: git pull, terraform init => boom

boom message (reproducile, similar messages…):

│ Terraform failed to fetch the requested providers for linux_arm64 in order to calculate their checksums: some providers could not be installed:
│ - registry.terraform.io/datadog/datadog: the current package for registry.terraform.io/datadog/datadog 3.3.0 doesn't match any of the checksums previously recorded in the dependency lock file
│ - registry.terraform.io/hashicorp/aws: context canceled.

The only work around I found so far is:

rm .terraform.lock.hcl
terraform provider lock  -platform=linux_amd64 -platform=linux_arm64 -platform=darwin_amd64 -platform=darwin_arm64 -platform=windows_amd64

If a dev runs terraform provider lock -platform.... before pushing, it works.
Do we REALLY need this or could the lock file contain an OS/arch attribute so it does not try to compare a hash that will be wrong because this is the wrong OS/arch?

Am I doing something wrong or is it the expected behavior since 0.14?

6

Hi @ohmer,

First I want to note that this topic was something we were using during the beta period for Terraform v0.14 and I ought to have locked it once we released v0.14.0 final, since the design is now largely “locked in” for v1.0 and later under our compatibility promises.

With that said, it sounds like you have some non-default CLI configuration which is overriding Terraform’s default behavior of installing each provider directly from its origin registry and, at the same time, capturing all of the checksums signed by the original publisher to get coverage across all platforms.

In that case, it is indeed necessary to explicitly use terraform providers lock because without direct access to the origin registry Terraform can only record the checksum of the specific packages it downloaded; it has no way to get the other trusted checksums that the provider developer signed.

The terraform providers lock command exists as a compromise so that you can day-to-day use whichever non-default provider installation mechanism you’ve configured, but you can still explicitly ask Terraform to generate a more compehensive set of checksums as part of the process of adding a new provider or upgrading an existing one. Developers should need to do this only if they’re explicitly changing the providers for this configuration, because otherwise the already-stored checksums will remain valid.

I hope the above answers your direct question of what the expected behavior is and, in turn, a little about why Terraform behaves in that way. Since I ought to have locked this topic back at the v0.14 release I’m going to do that now, but if you have other questions about these behaviors then please do feel free to start a new topic about it, and I’m happy to talk some more about how the dependency lock mechanism works and how different custom provider installation options can potentially affect it.

2 Likes