Terraform plan shows objects have changed outside TF

Terraform
1

Good afternoon,

Terraform v1.0.2

We have a very simple module to provision EC2 AL2 instances. Upon executing a “terraform plan” immediately after “apply” TF is reporting out-of-band changes were made.

  # module.sandbox.aws_instance.this[0] has been changed
  ~ resource "aws_instance" "this" {
        id                                   = "<id>"
        tags                                 = {
            "Role"        = "sandbox"
            "Terraform"   = "true"
        }
        # (29 unchanged attributes hidden)

      ~ root_block_device {
          + tags                  = {}
            # (9 unchanged attributes hidden)
        }
        # (4 unchanged blocks hidden)
    }

Unless you have made equivalent changes to your configuration,
or ignored the relevant attributes using ignore_changes, the
following plan may include actions to undo or respond to these
changes.

───────────────────────────────────────────

No changes. Your infrastructure matches the configuration.

Can someone explain what is happening here and how to prevent it? I should note we do NOT set tags on the root block device.

Thank you,
-Joef

2

Hi @Kimmel,

From the surface behavior here (and without looking at the provider implementation) this seems to be an inconsistency in the provider’s modelling of the situation where the block device has no tags.

Specifically, it looks like you wrote a configuration that didn’t set tags at all, and the provider responded to the “create” action by echoing back that unset tags argument. But then during the next refresh, the provider changed its mind and decided to signal “no tags” by writing an empty map instead.

If I’ve understood the cause correctly, I expect you could work around it by explicitly setting tags = {} in your configuration, so that the initial state of the object after creation will therefore agree with the result of refreshing.

The underlying problem here is related to a provider’s responsibility to distinguish between normalization and drift. For lots of remote systems there are many ways to physically represent the same logical result, and often in that case when we write something to the remote API and then read it back later the API will return it in some normalized form that might differ from what we originally sent. In this case it seems like the remote API doesn’t really distinguish between setting this argument to an empty map vs. not setting it at all, but both of those cases get “normalized” to an empty map when reading back.

The intended model here is that when a provider detects a difference it should classify it as either normalization vs. drift and only propose a change for the situation where it’s drift, but some less-frequently-arising situations can unfortunately get missed until someone encounters them. I think this particular example might also run into some trouble with the current Terraform SDK, whose API design can make it hard for provider code to distinguish between something being unset and something being set to the “empty” value of that same type.

With that said, it might be worth reporting this as an issue in the provider repository, although in practice I think the provider development team may be constrained in how they can resolve it directly with the current way this resource type is implemented, and so this rough edge might need to wait for some improvements to the underlying Terraform SDK. Either way, the provider development team will be in a better position to determine the path to a fix for this bug, which could then allow you to remove the workaround of explicitly setting tags = {} in your configuration.

3

Appreciate the thorough reply. I now better understand the potential mechanisms that could result in plan reporting the above.

I believe I tried setting “tags = {}” in my config, but I’ll need to validate that again.