Iterating help needed: map of lists, or list of maps, for subnet cidr+name matching?

#1

Hello everyone, and thanks in advance for any assistance you may be able to lend. I am just beating my head against a wall here. I want to avoid using a simple list to define subnets, because then I need to add a new resource block every time I build a new VPC with a different subnet name. For example:

With that example (taken more or less whole-cloth from the “terraform-aws-modules/vpc/aws” module), if I want to create a new subnet named “foo”, I need to build a whole new ‘resource’ block that iterates over a whole new list called “cidr_list_foo”, and it’s just… messy.

I want to be able to define a tree that matches my preferred subnet name to the various cidr_blocks associated with those subnets. For example, I’d like the end result to be as if I had defined my subnets with standard Terraform primitives like so:

But I don’t want to have to ‘unroll’ a loop like that. I don’t know what an appropriate data type would be, but I’m thinking it’s going to look something like a “list of maps” or a “map of lists”. If that’s the case, I just don’t know how to properly iterate over it. A “map of lists” might look something like:

mapped_subnets = {
    mgmt = {
      subnets = ["10.105.160.0/26", "10.105.160.64/26", "10.105.160.128/26", "10.105.160.192/26"]
    },
    app = {
      subnets = ["10.105.161.0/27", "10.105.161.32/27", "10.105.161.64/27", "10.105.161.96/27"]
    }
  }

whereas a “list of maps” would be:

mapped_subnets = [
{      
      name = "mgmt" 
      subnets = ["10.105.160.0/26", "10.105.160.64/26", "10.105.160.128/26", "10.105.160.192/26"]
    },
     {
      name = "app"
      subnets = ["10.105.161.0/27", "10.105.161.32/27", "10.105.161.64/27", "10.105.161.96/27"]
    }
  ]

Either way, I suspect I’m barking up the wrong tree because I can’t get the iteration right. If “for_each()” could be nested, this would be straightforward, but it can’t so it’s not. Any thoughts on how I can better structure my data, and then iterate over it, so I can be just a bit more dynamic?

#2

I try to stay away from lists as much as possible these days, simply because of day-two effects.

If you use a list to create subnets, then they get numbered 1 2 3, typcially there’s a [count.index] somewhere. And then if you remove one of them, all the subsequent subnets will be destroyed and recreated with their new number. That’s painful.

IF possible, I prefer to calculate subnet cidrs from the VPC/VNET using the cidrsubnet function so different environments using the same network structure are easy to create.

1 Like
#3

Ooof, cidrsubnet is awkward. I can see this being inherited by some $jr_admin down the road, them not fully understanding subnet/CIDR math, and all hell breaking loose. I suspect the best course of action is just for me to ‘unroll the loop’ then, and keep it within the basic primitives.

Thanks for cluing me in to the problem with lists. That, too, would get real ‘interesting’ (for Chinese versions of ‘interesting’) real fast.

#4

Hi @law,

In principle you can use any data structure as input as long as you can write an expression to transform it into a flat map to pass to for_each. The most common strategy for that is to use the flatten function in conjunction with for expressions.

Taking your mapped_subnets variable, for example:

variable "mapped_subnets" {
  type = set(object({
    name    = string
    subnets = set(string)
  }))
}

locals {
  flat_subnets = toset(flatten([
    for group in var.mapped_subnets : [
      for cidr in group.subnets : {
        group_name = group.name
        cidr_block = cidr
      }
    ]
  ]))
}

The above reorganizes the input data structure so that there is one object per distinct subnet, with each item looking like this:

{
  group_name = "mgmt"
  cidr_block = "10.105.150.0/26"
}

That result can then be transformed one more time into a map to make it compatible with for_each, using the group name and cidr block together to make a unique identifier for each subnet instance:

resource "aws_subnet" "example" {
  for_each = {
    for s in local.flat_subnets : "${s.group_name} ${s.cidr_block}" => s
  }

  vpc_id            = local.vpc_id
  cidr_block        = each.value
  availability_zone = "us-east-1a"

  tags = {
    Name = "${each.value.group_name} ${each.value.cidr_block}"
  }
}

In practice you’ll presumably have other details that must vary per subnet instance, such as availability_zone. So you’ll presumably want to change that var.mapped_subnets structure to include availability zones too, perhaps making the subnets object be map(string) instead so it can be a map from availability zone name to CIDR prefix:

variable "mapped_subnets" {
  type = set(object({
    name    = string
    subnets = map(string)
  }))
}

locals {
  flat_subnets = toset(flatten([
    for group in var.mapped_subnets : [
      for az, cidr in group.subnets : {
        group_name        = group.name
        availability_zone = az
        cidr_block        = cidr
      }
    ]
  ]))
}
#5

Yeah I know that cidrsubnet may be considered a bit esoteric, so I actually offer 3 different approaches in my module and then just merge the maps at the end. So you can hardcode cidrs, you can use cidrtsubnet - or simply provide a list of names and then everything happens automagically.

“What if somebody does X or Y in the wrong way?” - well then, you are supposed to look at the plan before you tear apart the production environment…

1 Like