Using default_tags {}

Greeting Terraform Community,
I am super excited to join the Hashicorp discussion space especially Terraform. My experience about Terraform between beginner and intermediate level. I had a question to ask so any help will be appreciated. I had created default_tags {} at was provider level. When I run “terraform plan” it runs without issue. But when I tried to run “terrafom apply” it returns

Error: Provider produced an inconsistent final plan

│ When expanding the plan for aws_subnet.terraform_subnet2 to include new values learned so far during apply, provider “registry.terraform.io/hashicorp/aws” produced
│ an invalid new value for .tags_all: new element “AZ” has appeared.

│ This is a bug in the provider, which should be reported in the provider’s own issue tracker.

it gives this unexpected error in different resources when trying to add more tags depends on the resource itself.
I am using
Terraform v0.15.1
on linux_amd64

so in theory the default_tags supported in the terraform version and AWS provider as well. Anyone who can help or advise will be appreciated.

Best regards

Just an update. When I tried to rerun it again it works. I destroyed and reapply again without changing any it fails like the original post

Hi @AhmedELShafaie,

It would be helpful if you could share the provider configuration you were using (the one including default_tags) and the configuration for aws_subnet.terraform_subnet2 to understand what exactly happened here.

You mentioned “when trying to add more tags depends on the resource itself” which makes me think that perhaps you were trying something like this:

provider "aws" {
  # ...

  default_tags = {
    AZ = aws_instance.example.availability_zone
  }
}

Using aws_instance specifically isn’t the point here, but rather populating that value based on any attribute that a provider will populate only after apply.

The problem here then is that it creates a situation where the provider configuration will be different at plan time than at apply time, and so the provider will end up producing a different set of tags and thus potentially run into this error.

This sort of problem is why the provider configuration docs say the following:

You can use expressions in the values of these configuration arguments, but can only reference values that are known before the configuration is applied. This means you can safely reference input variables, but not attributes exported by resources (with an exception for resource arguments that are specified directly in the configuration).

The fact that it worked on a subsequent plan seems to confirm this theory, because in that case the necessary object would already exist during planning and so the provider would already have access to the value of AZ at planning time. Allowing references to unknown values in provider configurations (rather than treating them as a hard validation error) was a historical design mistake which we’ve unfortunately had to preserve compatibility with because folks have intentionally built workflows around doing a two-step apply similar to what you tried here, and so we need to keep those working.

I’d suggest trying to find a different way to set your default_tags so that they depend only on values that are defined directly in your configuration.

If that isn’t possible then you can also potentially avoid the problem by using the -target planning option only for your initial creation, to make sure that the object which defines the availability zone will be created before planning anything else. For my example above that would look like this:

# First, apply only up to the EC2 instance to
# determine the availability zone.
terraform apply -target=aws_instance.example
# After that, do a normal apply to create
# everything else
terraform apply

The above is the sort of two-step workflow that we’re remaining compatible with by not rejecting unknown values in provider configurations. I would suggest treating it as a last resort, but if you do decide to go with it then the nice thing is that you only need to do that exceptional -target=... command once for initial bootstrapping, as long as you never recreate whichever object is determining the AZ tag value.

I’ll also add that this particular situation is, unfortunately, not something that provider developers are currently empowered to solve in spite of what that error message says, because provider-specific code doesn’t currently have access to information about whether an argument is “known” or not: that information gets stripped out by the SDK layer and replaced by a null value, which is why the AWS provider here thinks that you’ve not set AZ at all in that case.

Hopefully a future version of the Terraform SDK will make it possible for provider developers to distinguish unknown vs. unset values in the provider configuration, in which case the provider could then in principle generate an explicit error for this case, saying that for default_tags in particular all of the values must be known in order for that argument to be valid, so that this particular case could be validated even though we’re blocked by backward-compatibility from making that a general rule for all provider configuration arguments.

You could still report it as a provider bug just to record that the problem exists in a place that’s easier to refer to, but I expect that the provider team would respond to it by sending a request upstream to the SDK team to give the provider access to the information required to handle it properly.