In one our use-cases, we remove a VM instance (using terraform destroy with -target option) and also remove a few output variables related to that VM instance from the TF configuration. After that, we call the terraform refresh. Apparently, the output variables are not removed from Terraform state, even after refresh. (The removed resource does get removed from the state). Also tried to use “terraform state rm” to remove the output variables but that also doesn’t seem to work.
In Terraform 11, terraform refresh used to clean out the output variables in the state too.
Is this behaviour expected, or is it a bug? What do you suggest we should do in this case to remove the output variables from the state ?
It’s hard to be sure what exactly is happening here without seeing a working example, but unfortunately your approach here includes a combination of two different features that are not part of a typical recommended Terraform workflow, both of which have some design compromises to work around the fact that they violate some of Terraform’s typical assumptions:
For -target, you can end up in situations that would be impossible to reach via the normal workflow of editing the configuration and running terraform apply, and Terraform responds to those in a best-effort sort of way.
For terraform refresh, modern Terraform implements this just as creating a plan and then discarding any planned changes to resource instances before it saves the updated state, and so Terraform will typically avoid removing things that a terraform plan would’ve planned to create.
To avoid these oddities, I think your best option would be to use Terraform in the “normal” way:
Remove the VM instance you don’t want from the configuration, either by deleting its resource block or by using the count or for_each meta-arguments to cause the resource block to create zero instances.
Remove the output values that depend on it from the configuration, or write them as conditional expressions that return null when the corresponding resource isn’t being created.
Run terraform apply so Terraform can react to that change by deleting the resource and updating the output value.
Here’s an example of what that might look like in a relatively-simple case where you’ll either have zero or one instances based on a variable:
With the above example configuration, you’d destroy an existing object associated with aws_instance.example by setting the enable_instance variable to false when you run terraform apply. If you’d previously created that instance then Terraform will plan to destroy it in order to match the updated configuration.
There are various other ways to get a similar result, but the general point here is that -target is only for exceptional use in special situations where it’s documented as a workaround, and that anything you are doing routinely should be done using terraform plan and/or terraform apply in order to get a consistent result.
Thank you for the detailed reply !
Our main concern with “apply” for such changes has been that user might have done other things with the VMs after they were created with Terraform (e.g. attached other block volumes, added extra VNICs etc). “Plan” and “apply” so far as I understand, will potentially remove such changes because they are not present in the TF configuration or TF state. So plan and apply will become potentially dangerous operations in this situation. Or am I wrong in my understanding?
With any configuration management tool (such as Terraform) users should not modify the managed resources via any other mechanism other than updating the code/tfvars. If changes are made outside of Terraform they will be reverted the next time Terraform apply is run, which is exactly what is supposed to happen.
While it can be necessary to make emergency changes they should always be reflected in the Terraform code ASAP to prevent them being undone.
The possible changes you mention sound like “normal changes” rather than things needed to fix an incident, so should always be being done via Terraform code changes and then an apply - Terraform code is the source of truth and should be maintained accordingly.
When Terraform creates a plan, part of that process is to read the latest data from the remote system in the same way that terraform refresh would.
The differences between terraform refresh and terraform apply then are just the following:
terraform refresh doesn’t have a separate plan and apply step, so you don’t get an opportunity to review what changes it’s going to make. In today’s Terraform (when v0.15 is about to be released) terraform apply also doesn’t show you explicitly what changes it detected while refreshing, but you do at least get to see what actions Terraform is planning in response to the changes it detected, and you can discard the plan in order to avoid committing the refreshed state to the backend.
The terraform refresh command discards any actions that were planned against resources, so Terraform won’t plan any actions to change the objects back to match the configuration or to match an updated configuration.
If you are intentionally changing objects outside of Terraform and want to avoid Terraform undoing those changes then you can potentially accommodate that sort of workflow using ignore_changes, although I would consider that to be a last resort because it weakens Terraform’s usual goal of making remote objects converge on matching the configuration.