Updating tags passed to a terraform module results in a recreate!

Terraform
1

I’m running against an issue where updating tags (which I would consider to be an update in place operation) after deployment to at least 20+ aws account results in a module destroy and recreate. This is a huge problem as there are complex modules deploying GWLB 3rd party firewalls which would be very disruptive to an environment where all that is needed is to adjust billing allocation tags on a bunch of objects in the account. Changing tags on non-module code results in the expected in-place non-disruptive change. All modules tag changes are resulting in a destroy and create operation, very bad! Is there anyway to avoid this behavior, i’m really stuck now with something very unexpected?

Here is example in the modules which is identical to non-module code:
In the main code i have a var.tags list object with a whole bunch of tags read from an MS Devops custom pipeline. It’s always passed as a tag merge to the objects created. Updating tags works (in place update) on non-module objects, but passing new tags to a module and it merging the tags from the module variable passed results in a plan that indicates module needs to be destroyed and re-created.

variable "tags" {
  description = "Specifies object tags key and value. This applies to all resources created by this module."
  default = {
    "Terraform" = true
  }
}
resource "aws_route_table" "tg-publicnat" {
  count                     = 3
  vpc_id                    = var.tg_vpc.id
  tags = merge(
      var.tags,
      {
          Name                    = "tg-publicnat-az${count.index + 1}"
          "owner:creator:locked"  = "true"
      }
  )
}
2

Upon further investigation changing an existing tag value appears fine, the problem manifests when adding a new tag pair in the var.tags variable that ultimately get’s passed to the module.

3

Tracing all the dependencies in the plan, adding a tag is resulting the module indicating that subnets need to be replaced due to a data object that is returning the availability zones in the region. Very weird. How adding a tag value pair is resulting in an existing data element capturing the available availability zones in a region to be perceived by terraform plan as unknown or changed all of a sudden?

data "aws_availability_zones" "availability_zones" {}

availability_zone               = "us-west-2a" -> (known after apply) # forces replacement

example code in module creating a subnet that now Terraform believes needs to be destroyed and recreated due to adding a tag value pair from the var.tags variable:

resource "aws_subnet" "gwlb-endpoint1-az1" {
  vpc_id     = var.tg_vpc.id
  availability_zone = data.aws_availability_zones.availability_zones.names[0]
  cidr_block = "${cidrsubnet(var.tg_vpc.cidr_block, local.subnetting_coefficient, 6)}"
  map_public_ip_on_launch = "false"
  tags = merge(
      var.tags,
      {
          Name = "gwlb-endpoint1-az1"
          "owner:creator:locked"  = "true"
      }
  )
}
4

Problem worked around/resolved by passing the availabilityzones data element from the calling code to the module as a variable. original code had the module create subnets in 3 avail zone based on a local data.availabilityzone element.
Again not sure why adding tags all of a sudden is making the local data.availabilityzones be an unknown element requiring the subnets to be destroyed and recreated. But this work around is fine as far as I’m concerned.

5

Hi @mark.colatosti,

Unfortunately since I can only see fragments of what’s going on here I can’t really say for certain what’s going on, but a typical reason for a data resource read to be deferred to the apply step (which is the only reason why its results should be “(known after apply)”) is if there’s a dependency in the configuration forcing it to be handled only after a change to a managed resource.

I don’t see any depends_on in the code you’ve shared so far, but if for example the module block which called the module which includes that data "aws_availability_zones" "availability_zones" had depends_on set to refer to a managed resource, and that managed resource is changing to add new tags, then Terraform must delay reading the availability zones until the apply step in order to respect the dependency ordering that you’ve declared for that module.

If it seems like what I’ve suggested here might be the root cause of what you saw, then my suggested solution would be to avoid using depends_on in module blocks – it’s a very imprecise way to declare dependencies which therefore tends to cause far more dependency edges than strictly required – and instead define the required dependencies more precisely using references embedded in your expressions. I can’t be more specific than that without seeing what exactly the depends_on is currently intended to achieve, but hopefully since you can see the whole configuration yourself you can understand better than I can exactly what object inside the module needs to have the dependency, and declare it more precisely so that the data "aws_availability_zones" "availability_zones" block will be excluded from that dependency relationship. I’m happy to give more specific advice if you’re not sure but can say a little more about the underlying requirements.

6

I understand your comment about not being certain about root cause without seeing the entirety of the code, but its somewhat proprietary and quite long. I fortunately have no use of depends_on yet so it doesn’t exist anywhere in my code. Everything does seem to be pointing to a situation where a calculated data.availabilityzones element in the module is the root cause. This is used when creating 6 subnets and its these subnets that cause a trickle down replace on a number of elements. There appears to be no circular references or other problems. The issue goes away when creating the same subnets but this time passing the acvailability zones as a variable from the code calling the module, e.g.:

resource "aws_subnet" "gwlb-publicnat-az3" {
  vpc_id     = var.tg_vpc.id
  availability_zone = var.availability_zones.names[0]
  cidr_block = "${cidrsubnet(var.tg_vpc.cidr_block, local.subnetting_coefficient, 95)}"
  map_public_ip_on_launch = "false"
  tags = merge(
      var.tags,
      {
          Name = "gwlb-publicnat-az3"
          "owner:creator:locked"  = "true"
      }
  )
}

The only thing changing is using a var versus data element on the availability_zone parameter.