A Note On Breaking Changes In The Plugin SDK

Hey Terraform Provider Developers!

We’re the team responsible for maintaining the Terraform Plugin SDK, the framework you all use to write your Terraform providers. We’ve got some exciting news to share with all of you about what we’ve been working on, and some important information you need to know.

First and foremost, if you aren’t pinning to a specific version of the SDK using Go modules, you should start doing that today. Similarly, if you’re importing the SDK using github.com/hashicorp/terraform-plugin-sdk@master or github.com/hashicorp/terraform-plugin-sdk@latest, you should start using an actual release of the provider today.

We’re planning to, on Wednesday, move our development efforts for the upcoming v2.0.0 of the SDK to the master branch of our repo, and if you’re expecting the master branch to contain v1.X code, you’re going to get some unexpected breaking changes.

We’re planning to share a lot more details with you all about what changes we’re making in v2.0.0 of the SDK, but we’ll talk a bit more about them when we start releasing our release candidates. We’re excited about these changes, which we feel provide a nice balance of new functionality and backwards compatibility. We are hopeful that the process of upgrading to v2.0.0 will be relatively straightforward for most provider developers.

The Process

As breaking changes to our SDK are a relatively new thing, we wanted to outline what the process will look like for you all.

This post is our first step, a notification that we’re making breaking changes.

Next we’ll begin our code freeze and move our master branch to include the breaking changes. No breaking changes will be released at this time.

Once we have our breaking changes on master, we’ll begin testing those changes in an in-depth fashion on some providers. We test our changes on providers as we make them, and we’ve tested this release on various providers already, but we want to run some tests after code freeze to gain confidence in what we’re releasing.

When we’re satisfied with our testing, we’ll start shipping release candidates. Release candidates will be tagged pre-releases on our GitHub repo, and are open for public testing. We’d love if you’d upgrade your providers at this point and run tests, giving us feedback on any bugs that you encounter or any hardships you run into while upgrading.

We may have only the one release candidate, we may have multiple. We’ll endeavor to ship release candidates at a regular cadence, matching the rate of feedback we have for the release.

When we’re confident we’ve gotten enough information out of our release candidates, we’ll ship v2.0.0. At this point we consider it production-ready and encourage providers to update and adopt it as soon as reasonably possible.

What You Can Do To Prepare

We understand that provider development is a busy job and that this will require extra planning and work on your part. We believe the improvements we are delivering will be worth the extra planning and work, and hope you’ll agree.

To prepare for this process, there are a few things you can do today:

  • If you are importing github.com/hashicorp/terraform instead of github.com/hashicorp/terraform-plugin-sdk in your provider, you should upgrade to the standalone version of the SDK before trying to adopt v2. If you’re not using the latest released version of v1 of the SDK, we recommend upgrading to that now. This will help minimize changes when upgrading to v2, which will help us pinpoint bugs.
  • If you’re not part of this forum, we recommend joining it. We’ll be using this forum as a way to broadcast information to provider developers and communicate about future SDK developments.

As always, please feel free to let us know if you have any questions!


Hey Terraform Provider Developers!

Just as an update on this topic, we got feedback that getting a bit more information would be helpful in preparing for upgrading to version 2.0.0 of the SDK. Specifically, people are trying to gauge how disruptive the breaking changes we intend to make are, and how much time they should allocate for the upgrade process, along with what new features we’re introducing.

Because of the many layers of compatibility shims and the variation in the usage of the SDK, we sometimes need to modify, postpone, or cancel changes to the SDK, even late in their development cycles. We want to caution that this list is, by nature, incomplete and subject to change. It is the latest information we are working with, but some of these changes are still being made, and remain subject to more thorough testing. We hope it will help you understand the scope of version 2.0.0, and will help you plan, but you’ll forgive us if the final release does not match this list exactly.

Breaking Changes

There are a handful of breaking changes that we anticipate will affect only a small percentage of developers, mostly revolving around built-in helper functions. To keep this post reasonably-sized, we’re going to limit ourselves to changes we believe will affect the majority of developers.

Dropping Support For Terraform 0.11

The largest breaking change providers will need to be aware of is that v2.0.0 of the SDK will not support Terraform 0.11 or before. Terraform 0.12 or higher must be used with providers built against versions of the SDK after v2.0.0. Our upgrade guide will include information about how to make sure Terraform 0.11 and below do not accidentally download these versions of the providers when running terraform init.

0.12 introduced a large number of new ideas and features to Terraform, and thus far the SDK has only adopted those that could live side-by-side with 0.11. Unfortunately, a large number of improvements we would like to make to the SDK cannot be made while maintaining support for 0.11 and below. We’ve decided to focus on bringing more features to the SDK, instead of maintaining compatibility with 0.11 and before.

If your provider has not already ended support for Terraform 0.11, HashiCorp recommends you ship a new major version of your provider when dropping support.

See our blog post for more background on this change.

Removing Partial State

The ResourceData.SetPartial and ResourceData.Partial methods in the helper/schema packages were poorly understood and usually unnecessary. Most developers will just need to remove them from their code; in incredibly rare cases, a developer may need to update a resource’s design. Guidance on when either method was actually necessary will be included in our upgrade guide.

Panic On ResourceData.Set Errors

Previously, ResourceData.Set would return an error when trying to set data that doesn’t fit in the schema for the specified field. These issues were almost entirely programming errors, not usage errors. As such, these errors were routinely not checked, and would lead to unfortunate bugs from developer typos or type errors. We introduced a TF_SCHEMA_PANIC_ON_ERROR environment variable years ago that, when set, would cause the SDK to panic instead of returning an error. This was widely used in CI and helped prevent a number of bugs from making it to releases. With this experience, we’re now confident this is the correct default behavior for the SDK, and have updated it accordingly.

Now when ResourceData.Set would have returned an error, it will panic instead, unless TF_SCHEMA_PANIC_ON_ERROR is explicitly set by the user to a falsey value like 0 or false.

Deprecate Exists

While current code won’t break, we want to draw developers’ attention to the Exists function for schema.Resources. It is now deprecated, and we recommend moving its logic into your Read functions, instead. We found most of this logic got duplicated and run twice, anyways, for no gain. We’ll include instructions in our upgrade guide on how to move Exist functions into Read functions with minimal effort.

Context-Aware Functions

Under v2.0.0 of the SDK, the SDK accepts context.Contexts in a lot more functions and methods, which enables easier usage of defined timeouts and cancellation in providers. Most functions–like the Create, Read, Update, and Destroy functions that are the backbone of providers–are merely deprecated, with context-aware variants being available. Upgrading to the new context-aware versions of these functions should require only changing the name of the property the function is set on, and adding a context.Context to the parameters for the function. But they also do not need to happen immediately; the deprecated versions will still exist and work, we just recommend migrating off them as soon as it’s convenient, as they’re not guaranteed to stick around forever.

Some helper functions–most notably, CustomizeDiff–do not and cannot have this backwards compatibility available to them. They will require the context.Context to be added to their type signature when upgrading.

A Cleaner API Surface Area

A great many functionally internal methods, types, variables, and packages were removed from our public API–either deleted outright, unexported, or moved into an unimportable internal/ package. These parts of the API surface were never meant for consumption by provider developers, and we took the opportunity of a breaking change to remove them from the API, helping to enforce that expectation and more clearly communicate it.

New Features

With these breaking changes, we’re bringing a number of quality of life improvements to the SDK. This post is too long already, so we won’t cover all of them, but we want to talk about some of the main things we think people are looking forward to.

Diagnostics Support

The opt-in context-aware versions of the Create, Read, Update, and Destroy functions that providers rely on return a new Diagnostics type instead of an error. While using this new feature will require a small amount of work–we expect much of it to be search and replace, and will have more information in the upgrade guide–it enables a long-standing request and improves end user UX considerably. First, providers can now return warnings from Create, Read, Update, and Destroy functions. Second, errors (and now warnings!) that a provider generates will be more narrowly targeted at the exact field, sometimes even the exact item in a field, that prompted that feedback.

This is also automatically applied to validations using ValidateFunc in providers that upgrade, giving more precise code locations of validation issues.

More Accurate Test Runs

Previously, our test framework had a mocked copy of Terraform’s core logic that would simulate a Terraform run. As with all simulations, it was an imperfect-but-passable approximation of what Terraform actually did. In v2.0.0 of the SDK, we’ve modified the test framework so now tests run an actual Terraform binary behind the scenes. This ensures that the behavior you see in test runs matches the behavior your users will see in production, and it also opens up exciting possibilities for running tests against various versions of Terraform. Testing is obviously a crucial piece of Terraform provider development, and something we want to be careful about, so we’re still finalising details on this, but we anticipate it being largely a drop-in replacement for existing testing code. Notably, developers that manually check the values of items in a set in their tests may need to change those specific checks.

Feedback Welcome

These are the major features and breaking changes we anticipate making that we expect to have the most impact on provider developers. It is not exhaustive, and it’s subject to change, but it’s the latest information we’re working with. If you’d like to get a more detailed picture of what this all looks like for your provider specifically, we recommend running a test run of your provider after changing your SDK dependency to point to the master branch.

We’re always happy to receive feedback, and welcome any bug reports. Feedback about the process or that doesn’t have an easily defined “done” state it would be awesome to centralize here on the forums; bug reports are encouraged, as always, in the issue tracker.

Happy Terraforming!