Solve sorting problem

Terraform
1

Hello,

I had a problem with creation of multiple instances of one resource and setting dynamic attributes in another. Now it seems I solved one problem, but another remains.

Variables look like this:

variable "attached_vpcs" {
  description = "Map of attached VPCs"
  type        = map(map(string))
  default = {
    "vpc-0a1b2c3d4e5f" = {
      account            = "aws-account-1"
      cidrs_to_advertise = "10.0.0.0/22"
      security_domain    = "account-1-domain"
    },
    "vpc-01b2ac3d31f5" = {
      account            = "aws-account-2"
      cidrs_to_advertise = "10.1.0.0/22"
      security_domain    = "account-2-domain"
    },
    "vpc-03a1234567cd" = {
      account            = "account-3"
      cidrs_to_advertise = "10.2.0.0/22"
      security_domain    = "account-3-domain"
  } }
}

locals {
  vpc_ids          = keys(var.attached_vpcs)
  security_domains = [for id in local.vpc_ids : var.attached_vpcs[id].security_domain]
}

And the code is:

resource "aviatrix_aws_tgw" "tgw_one" {
  account_name = var.tgw_aws_account_name

  aws_side_as_number    = var.local_as
  manage_vpc_attachment = false
  region                = var.region
  tgw_name              = var.tgw_name

  security_domains {
    connected_domains = [
      "Default_Domain",
      "Shared_Service_Domain"
    ]
    security_domain_name = "Aviatrix_Edge_Domain"
  }

  security_domains {
    connected_domains = flatten(["Aviatrix_Edge_Domain", "Shared_Service_Domain", local.security_domains])

    security_domain_name = "Default_Domain"
  }

  security_domains {
    connected_domains = [
      "Aviatrix_Edge_Domain",
      "Default_Domain"
    ]
    security_domain_name = "Shared_Service_Domain"
  }

  dynamic "security_domains" {
    for_each = local.security_domains
    iterator = domain
    content {
      connected_domains = [
        "Default_Domain"
      ]
      security_domain_name = domain.value
    }

}
}

resource "aviatrix_aws_tgw_vpc_attachment" "vpc_attach" {
  for_each                       = var.attached_vpcs
  tgw_name                    = aviatrix_aws_tgw.tgw_one.tgw_name
  region                           = var.region
  security_domain_name        = var.attached_vpcs[each.key]["security_domain"]
  vpc_account_name               = var.attached_vpcs[each.key]["account"]
  vpc_id                                     = each.key
  customized_routes              = lookup(var.attached_vpcs[each.key], "cidrs_to_propagate", "")
  customized_route_advertisement = lookup(var.attached_vpcs[each.key], "cidrs_to_advertise", "")
}

  1. We use for_each construction in the bottom resource to eliminate the problem of recreating instances when a source list changes and indices break. Now it’s a map with VPC IDs as unique keys. Adding/removing values from the variable is now correctly impacting only relevant instances.

  2. For the first resource though, the flatten() function and dynamic block with for_each both take the local.security_domains list, that gets created with the help of local.vpc_ids. And this vpc_ids has keys() function involved that sorts VPC IDs lexicographically.
    As a result, every time we add or remove items in var.attached_vpcs, the product of flatten() is reshuffled, and the entire order of dynamic “security_domain” arguments is changed.

The plan would look like this:

~ security_domains {
           aviatrix_firewall    = false
         ~ connected_domains    = [
               "Aviatrix_Edge_Domain",
               "Shared_Service_Domain",
             - "account-1-domain",
               "account-2-domain",
               "account-3-domain",
             + "account-1-domain",
           ]
           native_egress        = false
           native_firewall      = false
           security_domain_name = "Default_Domain"
       }


       ~ security_domains {
             aviatrix_firewall    = false
             connected_domains    = [
                 "Default_Domain",
             ]
             native_egress        = false
             native_firewall      = false
           ~ security_domain_name = "account-1-domain" -> "account-2-domain"
         }

         ~ security_domains {
               aviatrix_firewall    = false
               connected_domains    = [
                   "Default_Domain",
               ]
               native_egress        = false
               native_firewall      = false
             ~ security_domain_name = "account-2-domain" -> "account-3-domain"
           }
           ~ security_domains {
                 aviatrix_firewall    = false
                 connected_domains    = [
                     "Default_Domain",
                 ]
                 native_egress        = false
                 native_firewall      = false
               ~ security_domain_name = "account-3-domain" -> "account-1-domain"
             }

However, the Aviatrix provider doesn’t allow renaming security domains, and would destroy/recreate them, and this is impossible without detaching VPCs, basically makes the operation of adding another VPC impactful for the existing clients.

We tried to use set() type instead of map() to avoid implicit sorting, but every for: and for_each: operation still does some ordering in the background.

We were thinking of using separate simpler variables, maybe with some redundancy:
var.vpc_ids (list(strings)) +. var.security_domains (object(vpc_id => string)), and later do zipmap(), but kinda stuck there too.

Please recommend the way of maintaining unsorted values, so each time we append to the var.attached_vpcs, the dynamic block would only add a new argument to the end, and flatten() would also append to the existing list.
Thank you.

1 Like
2

Hi @mshakhmaykin,

Unfortunately, the handling of the ordering of nested blocks inside a resource type is decided by the provider in its schema, and is not something you can influence by configuration.

I can see in the source code of this provider that security_domains is currently declared as being a list, which means that the provider is telling Terraform that the ordering of these blocks is significant and so a change of ordering is therefore also significant.

I can’t think of a way to get the result you were looking for – treating the security domains as unordered – without the provider itself changing that block to be represented as a schema.TypeSet value instead. That would cause both the provider and Terraform to track each block independently by its contents rather than tracking them by their position in the ordering, and so the order you write them (or dynamically generate them) in configuration would not be significant.

Whether that change to the provider would be appropriate will depend on how the underlying Aviatrix API treats these objects; if they are unordered in the underlying API then switching to a set is likely to be the most accurate translation of the remote schema, but if they are ordered in the underlying API then the provider must treat them as ordered too.

I’m not familiar with the Aviatrix API at all so I’m not sure whether this change would be appropriate, but maybe you could open an issue in the Aviatrix provider repository to share the challenge you’ve encountered and see if there is a possible change to the provider that could better support your use-case.

1 Like