Terraform 0.14: The Dependency Lock File

Ever since Terraform has had support for installing external dependencies (first external modules in early Terraform, and then separately-released providers in Terraform 0.10) it has used a hidden directory under .terraform as a sort of local, directory-specific “lock” of the selected versions of those dependencies. If you ran terraform init again in the same working directory then Terraform would select those same versions again.

Unfortunately this strategy hasn’t been sufficient for today’s most common usage patterns for Terraform: many people on a team all using Terraform together, either on their separate workstations (so each has their own working directory) or in an ephemeral remote execution environment (so an entirely new working directory is created for each run).

Terraform 0.14 aims to address this by introducing a new model for tracking dependency selections: a new file in your root module which you can include in your version control system so that you can communicate about changes to dependency versions in the same way as you might communicate about changes to your main configuration: through a code review process.

The first iteration of this for Terraform 0.14 covers only provider dependencies. Tracking version selections for external modules will hopefully follow in a later release, but requires some deeper design due to Terraform’s support for a large number of different installation methods for external modules.

When you run terraform init for the first time after upgrading to Terraform 0.14, Terraform will generate a new file .terraform.lock.hcl in the current working directory, alongside the existing local cache directory, .terraform. This new dependency lock file is intended to retain metadata about the external items that Terraform will fetch into the .terraform directory, so that you can include the lock file in your version control while continuing to exclude the .terraform directory.

Once the lock file is generated, terraform init can make two additional guarantees on future runs:

  • It will always select the same version of each provider, even if a newer release becomes available that would also be accepted by the configured version constraints.

    This separates the idea of which provider versions a module/configuration is compatible with (the version constraints) from which versions a configuration is currently using (the lock file).

  • It will check the downloaded provider packages against the checksums it saw the first time you saw the provider, raising an error if any package does not match.

    This therefore implements a “trust on first use” model where you can, if you wish, perform audits or other checks on new provider versions you intend to use and then use the lock file to remember the checksums of the packages you audited, so you’ll know if the upstream provider package is modified in some way after you reviewed it.

You can run terraform init -upgrade to turn off the new rules described above and have Terraform instead select the newest available version of each provider that matches the version constraints. In that case Terraform will still update the dependency lock file to match the new selections, so you can use your version control system to review the changes to that file and decide if you want to keep those new selections.

As much as possible we’ve aimed to keep the new behavior consistent with the old behavior if we imagine the contents of the .terraform directory as formerly being a sort of implied “lock file” that survived only as long as the working directory it was created in.

The new dependency lock file formalizes that behavior by recording the same information in a file you can easily keep under version control, and so the lock file is now the authority for what is selected for a particular configuration, and the .terraform directory is in turn just a local cache of remote the items the lock file describes.

Although the dependency lock file is primarily an artifact maintained automatically by Terraform itself, we’ve used the familiar HCL syntax for it because we expect teams to see diffs of this file during their code review processes and we want those diffs to be easy to read and understand. The following is an example dependency lock file recording a single provider dependency:

provider "registry.terraform.io/hashicorp/azurerm" {
  version     = "2.1.0"
  constraints = "~> 2.1.0"
  hashes = [
    "h1:EOJImaEaVThWasdqnJjfYc6/P8N/MRAq1J7avx5ZbV4=",
    "zh:0015b491cf9151235e57e35ea6b89381098e61bd923f56dffc86026d58748880",
    "zh:4c5682ba1e0fc7e2e602d3f103af1638f868c31fe80cc1a884a97f6dad6e1c11",
    "zh:57bac885b108c91ade4a41590062309c832c9ab6bf6a68046161636fcaef1499",
    "zh:5810d48f574c0e363c969b3f45276369c8f0a35b34d6202fdfceb7b85b3ac597",
    "zh:5c6e37a44462b8662cf9bdd29ce30523712a45c27c5d4711738705be0785db41",
    "zh:64548940a3387aa3a752e709ee9eb9982fa820fe60eb60e5f212cc1d2c58549e",
    "zh:7f46749163da17330bbb5293dc825333c86304baa0a7c6256650ac536b4567c8",
    "zh:8f8970f2df75ac43ffdd112055ee069d8bd1030f7eb4367cc4cf494a1fa802c3",
    "zh:9ad693d00dc5d7d455d06faba70e716bce727c6706f7293288e87fd7956b8fe0",
    "zh:b6e3cb55e6aec62b47edd0d2bd5e14bd6a2bcfdac65930a6e9e819934734c57b",
    "zh:d6a3f3b9b05c28ecf3919e9e7afa185805a6d7442fc4b3eedba749c2731d1f0e",
    "zh:d81fb624a357c57c7ea457ce543d865b39b12f26c2edd58a2f7cd43326c91010",
  ]
}

The checksum verification feature works best if you do your initial installation from a provider’s origin registry, because Terraform can then record all of the checksums that were cryptographically signed by the original author. However, checksum verification is also supported for installation from filesystem and network mirrors, and Terraform 0.14 includes a new command terraform providers lock that may help you seed a lock file with official upstream checksums in preparation for later installation from your own mirror.


The dependency lock file mechanism will be included in all of the beta releases of Terraform 0.14, and we’d appreciate your early feedback so we can address any rough edges prior to the final 0.14.0 release.

Because the dependency lock behavior is centered primarily around the provider installation step in the terraform init command, you can test it against your existing configurations safely, as long as you don’t change your backend configuration and you don’t subsequently run any other commands that could modify your state. The terraform validate command is a good choice to run after terraform init to verify that Terraform is able to find and communicate with the installed plugins while avoiding any external side-effects.

After you have tested, we recommend that you discard the generated .terraform.lock.hcl file, rather than including it in your version control main branch, because feedback during the beta process may cause us to change the format of the lock file slightly before final release. Once the lock file format has been included in a stable release, new Terraform releases will remain compatible with it.

You may also need to delete the .terraform cache directory after you conclude your beta testing, because the file structure created in there by Terraform 0.14 may not be forward-compatible with all previous versions of Terraform. You can run terraform init again with your previous Terraform version to create a new .terraform cache directory.

We have some ideas for things to test listed below, but it’s also very helpful to try some situations we didn’t consider, in case there are opportunities to improve the behavior for less common situations.

  • Is terraform init able to correctly determine all of the providers your configuration depends on and record its version selections in the lock file? (Note: The lock file intentionally excludes the terraform.io/builtin/terraform provider because it’s always distributed as part of Terraform CLI itself, not as a separately-installed plugin.)
  • If you use either a local filesystem mirror or a network mirror to distribute providers to your Terraform execution environments, is terraform init still able to install from those mirrors and verify the package checksums against them?
    • Furthermore, are you able to use the terraform providers lock command to populate the lock file with the official checksums from each provider’s origin registry, and then have terraform init successfully verify your mirrored packages against those checksums?
  • If you change a version constraint in your configuration to permit a newer version (or remove the constraint entirely) after the lock file is populated, does terraform init with no arguments still install the previously-selected version? Can you then run terraform init -upgrade and see Terraform select the newer available version instead?
  • If you are in an environment where people run Terraform on different platforms, such as some folks using Windows and other folks using macOS, can you commit your dependency lock file to a temporary branch in your repository and then have a colleague who uses a different platform check out that branch and run terraform init? Does Terraform still successfully follow the selections recorded in the lock file, even though the second person needs to use a different build of the provider plugin?

If you have feedback to share, please feel free to reply to this topic! If some feedback spurs a longer discussion than I may split that discussion out into a separate topic so we can keep this one easier to follow, but starting here will make it easier to find all of the relevant feedback in one place.

Thanks!

2 Likes