Is it possible to dynamically pass query parameters in a Terraform resource’s Read method?
We operate an API that we’ve built a Terraform provider for. We recently introduced a v2 for an API, which essentially has v2 versions of all v1 resources. The v2 API’s READ method offers migration support in the form of `migrate` parameter, such that when it’s set to true, it will fetch the v1 resource, map it to its v2 representation and return the mapped rep.
I’d like to expose this functionality in our Terraform provider. The workflow I have in mind is: a a practitioner adds an `import` block and runs `terraform plan -generate-config-out=“generated_resources.tf”`. When doing the import, the provider would then pass the `migrate=true` query parameter to the v2 API, and unmarshal the mapped v2 response to the v2 resource model, and return the v2 resource’s Terraform configuration.
One idea would be to add an identity block to the v2 resources consisting of the resource’s id and its version (as per the API). Then a v1 API “resource” could be imported into a v2 Terraform resource by an identity block consisting of the v1 resource’s id and its v1 version. Does that seem plausible?
Hi @csar398,
If you are asking about a situation where your provider has a different resource type for “v2” than it does for “v1” then you might be able to use the State move functionality to get a better user experience for this.
Someone who was previously using your “v1” resource type can include a declaration like this:
moved {
from = example_v1.a
to = example_v2.a
}
…in which case if there’s already an object for example_v1.a tracked in the prior state, before calling “read” Terraform will first check if the provider knows how to translate a example_v1 object to the example_v2 schema, and then the logic in your provider could implement that by using the “migrate” feature you mentioned. If successful then Terraform will save the result as if it were the prior state for example_v2.a and then proceed with the normal read and planning calls.
Because this is “moving” rather than “importing” this does admittedly mean that there will be no automatic migration of the original source code: at the same time as adding the moved block, the author would also need to change their resource "example_v1" "a" block into a resource "example_v2" "a" block before running terraform plan or terraform apply, so that Terraform can understand that the intention is to migrate the existing object to that new configuration block.
If you think that the work to change an example_v1 config into an example_v2 config is too complex to be done manually, you could potentially offer your own tool to do that outside of the main Terraform workflow, since that’s usually where development tasks like editing code happen. But there isn’t any support for doing that automatically as part of the terraform plan command today.
Thanks for the great insight @apparentlymart.
But [`terraform plan -generate-config-out-path`]( terraform plan command reference | Terraform | HashiCorp Developer ) does support that (experimentally), doesn’t it?
I briefly considered implementing `ResourceWithMoveState` but discarded the option since I didn’t want to implement the v1 → v2 mapping in the provider or to dispatch it to an API call from the provider (read with query parameter migrate); I prefer to issue API calls only from within resources’ CRUD methods and keep them out of other provider lifecycle processes.
If I can verify there would be no side effects by calling the API from `MoveState`, I may implement it additionally (i haven’t worked with that yet, so I can’t tell). So a user could first just run `terraform plan -generate-config-out-path` to get the migrated resource’s config stub, and then replace the `import` block with a `moved` block targetting the generated (and possibly supplemented) config stub.
Let me summarize my idea for supporting the v1 to v2 api migration in our provider:
- Generate
example_v2 configuration for a given example_v1 resource:
- Move existing
example_v1 to example_v2 using the newly generated configuration:
- Implement
ResourceWithMoveState for example_v2
- The
MoveState function:
- Reads the
SourceState data into the v1 resource’s model.
- Calls the API’s read method using the v1 resource’s id (and
migrate=true set).
- Unmarshals the response into the v2 resource’s model, and writes that into the v2 resource’s
TargetState
- → This doesn’t work: The server’s
MoveResourceState method doesn’t configure the resource, in particular, no client is avaliable at this point.
Are there any hard requirements why MoveResourceState should be able to work offline with no access to an API client?
For us, the use-case is to be able to implement the attribute mappings involved in resource move just in the API, with no need for replication in the provider.
Hi @csar398,
When I first read what you wrote I incorrectly assumed that your “migrate from v1 to v2” was client-side code in your Go SDK rather than something that would require an API call. Indeed, this “state move” functionality is intended more for schema shape migration than it is for causing side-effects in the remote system.
If this “migration” operation has side-effects then I agree it’s probably best not to do it quietly as part of the state move functionality, but I have an idea (untested) for how you could still use “state move” as a part of the process.
The Terraform plugin protocol includes a mechanism for a provider to save a small amount of “private data” as part of the state of a resource instance, which you can use to keep track of metadata that is needed for the internal mechanisms of the provider that should not be exposed to Terraform configuration. From Terraform’s perspective this is just an arbitrary []byte value that Terraform just promises to provide back verbatim on the next call related to the same resource instance.
The “state move” API in the plugin framework exposes this private data through SourcePrivate in MoveStateRequest and TargetPrivate in MoveStateResponse, and so it’s possible for the provider logic to modify the private data at the same time as it’s doing the data transformation work.
In principle then, if the v1 and v2 representations are compatible enough you could implement “state move” as just an initial static data transformation implemented locally within the provider but also have it return a special private data value representing that it’s representing a partial migration from v1 to v2. Then when Terraform subsequently sends you the same object in Read or ModifyPlan you can check the private data to perform additional work to either perform the migration immediately or prepare to do it during the apply phase, depending on what seems most appropriate for how your migration functionality behaves.
If you choose to implement the final migration call only during the apply phase then there’s an extra step: the plan you return from a resource instance also has a similar idea of “private data” that’s separate from the private data of the state object, intended for use for internal information related to the planned change rather than to the prior object. So you can make ModifyPlan return some private data representing the need to perform migration during the apply phase, and then check for that in your apply-time code.
Since the private data is not actually visible to the end-user in the Terraform UI, if migration is an operation with significant consequences (e.g. if it would not be possible to migrate back to v1 afterwards) then I’d suggest also having ModifyPlan return a warning diagnostic which notes that a migration will be performed and describes anything the operator ought to know before approving the plan.
I hope that’s useful in some way! Terraform’s current design isn’t really optimized for implementing a migration path like this, but the mechanisms I’ve described here can be helpful for situations where you need to achieve something that’s a little outside of Terraform’s intended design.