Development overrides for providers under development

Terraform
1

Hi Mart,

I am feeling a little friction in my workflow with the addition of the lock file and want to let you know.

My situation is that I develop a provider locally and require it in my modules with:

terraform {
  required_providers {
    myprovider = {
      source  = "myprovider.com/myprovider/myprovider"
      version = "99.0.0"
    }
  }
}

Note the version is always 99.0.0 and my provider build script simply builds the binary and places it in the version 99.0.0 location on my local machine. With the addition of the lock file, anytime I rebuild the local provider and run terraform init will fail with:

Error while installing myprovider.com/myprovider/myprovider v99.0.0: the local
package for myprovider.com/myprovider/myprovider 99.0.0 doesn't match any of the
checksums previously recorded in the dependency lock file (this might be
because the available checksums are for packages targeting different
platforms)

Even with terraform init -upgrade I see the same error, presumably because the version has not changed so it still expects to see the same checksum. The workaround is simple enough: rm .terraform.lock.hcl though perhaps an option for init like -ignore-checksum-validation could also be useful in my case.

Just wanted to share my experience, please let me know if there is a better workflow for local provider development or any other comments!

Best – Cyrus

2

Hi @CyrusJavan,

I’m sorry to hear this new feature is conflicting with your existing workflow.

Indeed, the intent of the lock file is, in a sense, to prevent the very thing you are doing: once there’s a v99.0.0 of myprovider.com/myprovider/myprovider then that version is expected to be immutable and any later changes will be a different release.

However, we knew that since the new provider addressing approach in v0.13 it’s been awkward to do provider development, and so Terraform v0.14 also includes a new mechanism specifically for that use-case: Development Overrides.

This mechanism causes Terraform to ignore the published versions of a given provider altogether and instead to just use a specific local directory regardless of which version is selected. You can enable it in the CLI configuration file (which is not the same as .tf files) using a dev_overrides block within a provider_installation block:

provider_installation {
  dev_overrides {
    "myprovider.com/myprovider/myprovider" = "/path/to/directory"
  }
}

The directory you specify here must contain, in this case, an executable file with the prefix terraform-provider-myprovider, with a directory structure the same as you’d get when unzipping a published provider distribution package.

This mechanism is a bit different than the existing provider installation methods because it isn’t really an installation method at all. Instead, it directs Terraform to totally ignore what terraform init selected and use the given directory as the implementation of that particular provider instead.

Notice that the override isn’t versioned at all, because it’s the main Terraform operations – terraform plan, terraform apply, etc – which are paying attention to this setting, rather than terraform init. Because the provider installer is being ignored entirely here, the version number is irrelevant and version constraints in the configuration will only affect what terraform init does, not what the other commands do.

I hope this new mechanism will be more convenient for you than the workaround of publishing with a placeholder version number that you were following before.

3

Thanks for the help, this is exactly what I’ve been wanting!

Side question: Is there plans to have terraform init pay attention to the dev_overrides in the future (to avoid downloading a provider that is never used)?

4

Hi @CyrusJavan,

At the moment we’re waiting to see how the existing behavior is used before adding additional complexity. The current behavior is intended so that having a development override configured wouldn’t cause something to unexpectedly end up missing from the lock file (because terraform init didn’t have a chance to update it), but we may be able to make a different compromise later based on some practical experience with this new mechanism.

The good news is that for a typical “throwaway” small configuration written to test a provider there often aren’t any nested modules, any providers other than the one being tested, and no special backend configuration. As a result, you can typically just run terraform apply with the override enabled and skip running terraform init altogether, because nothing in the configuration depends on anything that terraform init would’ve done.

5

Interesting, I brute forced my way around this problem with a simple make target

tf: install
	rm -f examples/sequence_file/.terraform.lock.hcl
	terraform -chdir=examples/sequence_file init
	terraform -chdir=examples/sequence_file plan

Avoiding the init would speed up the iteration time though.

Based on just reading your comment and the linked documentation (I’ll try and find time to give it a try over the weekend) my concern with with the CLI configuration solution is that there is a risk of mistakes if not very careful when context switching between working on a provider and using terraform for practical work. For example;

  1. Configure CLI to use my local build
  2. Do some work creating a PR against existing provider
  3. Pager goes off/colleage/dev asks for assistance
  4. Run terraform against a real project
  5. ooops did I just use my local provider instead of the stable one?

I think I would feel safer if there was just some kind of no_lock parameter in the required_providers block, then I could turn off the lock for a simple test and maintain my safety for the day to day operations work.

6

I’m afraid I lost track of this topic along the way once the v0.14 beta period ended and we switched to looking for feedback in the more usual locations – sorry about that.

I expect this information may now be too late to be helpful, but just in case I’ll note that one pattern I’ve used and seen other folks use when working on a provider is to write a special CLI configuration file that’s used only for provider development. You can create a file with the suffix .tfrc anywhere on your system and then instruct Terraform to use it (overriding the usual search locations) by setting the TERRAFORM_CONFIG environment variable. For example, on a Unix system with bash:

$ export TERRAFORM_CONFIG=$HOME/foo-provider-dev.tfrc
$ terraform apply

You can then put the special dev_overrides setting only in that special file and thus it won’t be in scope for your day-to-date Terraform usage. When you begin a provider development session, you can temporarily configure just that one shell to have the override active, and then work with potentially multiple different test configurations where all of them would then use the overridden provider. Once you’re finished work on the provider, just exit that shell and the override will be forgotten until next time.

1 Like