How to iterate through nested objects/maps?

Terraform
1

Background:

Here is the map we are trying to iterate through:

cl_vnet     = {
  "xx.xx.1.0/24" = {
    subnets = {
     main = {
       application         = "app1",
       addr_prefix         = "xx.xx.1.0/25",
       service_endpoints   = [],
       service_delegations = {}
     },
     backend = {
       application         = "app1",
       addr_prefix         = "xx.xx.1.128/25",
       service_endpoints   = [],
       service_delegations = {} 
     }
    }
  }
}

In this block of code below, we are able to reference each subnet (main and backend) in the dynamic subnet block successfully.

    resource "azurerm_virtual_network" "spoke-vnet" {
  for_each = var.spoke_address_space

  name                = "${local.infra-prefix}-${local.region-code}-vnet" 
  location            = var.location
  address_space       = [each.key]
  resource_group_name = azurerm_resource_group.spoke-rg.name

  tags = local.tags

  dynamic "subnet" {
    **for_each = each.value.subnets**
    content {
      name = "infra-${var.sz_app_group}-${subnet.value.application}-${var.sz_environment}-${local.region-code}-${subnet.key}-subnet"
      address_prefix = subnet.value.addr_prefix
    }
  }
}

Result:

    # module.cl-spoke.azurerm_virtual_network.spoke-vnet["xx.xx.xxx.0/24"] will be created 
 + resource "azurerm_virtual_network" "spoke-vnet" { 
 + address_space = [ 
 + "xx.xx.xxx.0/24", 
 ] 
 + id = (known after apply) 
 + location = "northcentralus" 
 + name = "infra-cl-dev-ncus-vnet" 
 + resource_group_name = "infra-cl-dev-ncus-rg" 
 + tags = { 
 + "Application" = "cl" 
 + "BillingCode" = "04070-74400" 
 + "BusinessOwner" = "IT" 
 + "Environment" = "dev" 
 + "Group" = "Shared" 
 + "ReferenceNumber" = "1234" 
 } 
 
 + subnet { 
 + address_prefix = "xx.xx.xxx.0/25" 
 + id = (known after apply) 
 + name = "infra-cl-app1-dev-ncus-main-subnet" 
 } 
 + subnet { 
 + address_prefix = "xx.xx.xxx.128/25" 
 + id = (known after apply) 
 + name = "infra-cl-app1-dev-ncus-backend-subnet" 
 } 
 }

Now we want to iterate through the subnets again this this resource:

    resource "azurerm_subnet" "app-subnets" {
  **for_each = var.spoke_address_space.value.subnets**

  name                      = "infra-${each.value.application}-${var.sz_environment}-${local.region-code}-${each.key}-subnet"
  resource_group_name       = azurerm_virtual_network.spoke-vnet[var.spoke_address_space.key].resource_group_name
  virtual_network_name      = azurerm_virtual_network.spoke-vnet[var.spoke_address_space.key].name
  address_prefix            = each.value.addr_prefix

  dynamic "delegation" {
    for_each = each.value.service_delegations
    content {
      name = delegation.key
      service_delegation {
        name    = delegation.value.name
        actions = delegation.value.actions
      }
    }
  }
}

Result:

Error: Missing map element 
 
 on .terraform\modules\cl-spoke\main.tf line 57, in resource "azurerm_subnet" "app-subnets": 
 57: for_each = var.spoke_address_space.value.subnets 
 |---------------- 
 | var.spoke_address_space is map of object with 1 element 
 
This map does not have an element with the key "value".

What is the proper way to iterate through this map in the second resource?

1 Like
2

I assume your map is the value of the spoke_address_space variable. If so, it looks like you have a spurious .value in the for_each argument to the "app-subnets" resource:

for_each = var.spoke_address_space.value.subnets

I think this should be:

for_each = var.spoke_address_space.subnets

Does that make sense?

3

Hi @alisdair, I’m working with @spjavid on this issue.

When we run with the value of var.spoke_address_space.subnets, we get the following error:

    Error: Missing map element

  on .terraform\modules\cl-spoke\main.tf line 57, in resource "azurerm_subnet" "app-subnets":
  57:   for_each = var.spoke_address_space.subnets
    |----------------
    | var.spoke_address_space is map of object with 1 element

This map does not have an element with the key "subnets".

When we create the subnet as a part of the azurerm_virtual_network resource, we use each.value.subnets to obtain the properties of the subnets, but this does not seem to be the case for when we try to do the same thing in the azurerm_subnet resource.

edit: I should add that the dynamic subnet does not give us the subnet options we need of delegations and endpoints so that’s why we need to use the azurerm_subnet resource instead.

1 Like
4

Ah, I think I see what’s happening more clearly now. I misunderstood the map because its indentation is a little off.

For resource "azurerm_subnet" "app-subnets", you’re using for_each to create multiple resources. Which values in the nested map are you expecting to cause a resource to be created?

for_each can only iterate over a single level of map, so you may need to derive a local variable which is a flattened map of the objects you want to create resources for.

1 Like
5

Ah, the indentation got a little weird in the code block. @spjavid is fixing that in the OP.

Our idea is to have a map that contains all of the properties of the vnet, including the multiple subnets.

For azurerm_virtual_network we want to iterate through the first level of our variable, in which the key is the address space.

For subnets, we want to iterate through the subnet level of the map.

The variable would be like this:

vnet = {
  "xx.xx.1.0/24 = { # this is the first vnet we want to iterate through for the azurerm_virtual_network resource
    subnets = {
      first = { # this is the first subnet we want to iterate through for the azurerm_subnet resource
          application         = "app1",
          addr_prefix         = "xx.xx.1.0/25",
          service_endpoints   = [],
          service_delegations = {}
        }
      }
    }
  }
}
6

That makes sense to me. I think indeed you want to create a derived local variable, then. Here’s an isolated configuration for you to try out, using the merge function and the ... argument expansion operator:

variable "networks" {
  default = {
    "xx.xx.1.0/24" = {
      subnets = {
        main = {
          application         = "app1",
          addr_prefix         = "xx.xx.1.0/25",
          service_endpoints   = [],
          service_delegations = {}
        },
        backend = {
          application         = "app1",
          addr_prefix         = "xx.xx.1.128/25",
          service_endpoints   = [],
          service_delegations = {} 
        }
      }
    },
    "yy.yy.1.0/24" = {
      subnets = {
        main = {
          application         = "app2",
          addr_prefix         = "yy.yy.1.0/25",
          service_endpoints   = [],
          service_delegations = {}
        },
        backend = {
          application         = "app2",
          addr_prefix         = "yy.yy.1.128/25",
          service_endpoints   = [],
          service_delegations = {} 
        }
      }
    }
  }
}

locals {
  subnets = merge([
    for k, v in var.networks:
      { for sk, sv in v.subnets: "${k}-${sk}" => sv }
  ]...)
}

resource "null_resource" "network" {
  for_each = var.networks
}

resource "null_resource" "app-subnet" {
  for_each = local.subnets
}

This is the result:

$ terraform show
# null_resource.app-subnet["xx.xx.1.0/24-backend"]:
resource "null_resource" "app-subnet" {
    id = "3770721287261831912"
}

# null_resource.app-subnet["xx.xx.1.0/24-main"]:
resource "null_resource" "app-subnet" {
    id = "1876349133979128379"
}

# null_resource.app-subnet["yy.yy.1.0/24-backend"]:
resource "null_resource" "app-subnet" {
    id = "4007949681883369310"
}

# null_resource.app-subnet["yy.yy.1.0/24-main"]:
resource "null_resource" "app-subnet" {
    id = "7127906123058230762"
}

# null_resource.network["xx.xx.1.0/24"]:
resource "null_resource" "network" {
    id = "9187664947817107618"
}

# null_resource.network["yy.yy.1.0/24"]:
resource "null_resource" "network" {
    id = "5493479505369028810"
}

Note that I have barely any experience with the Azure provider, and this solution may not be right for your use case. I’m just trying to help with the configuration language—hope this makes sense!

1 Like
7

Thank you for the reply! Your answer makes sense. I have not looked into the merge function but we have begun to attempt using flatten as per this Stack Overflow question: https://stackoverflow.com/questions/58343258/iterate-over-nested-data-with-for-for-each-at-resource-level

It seems to be doing the trick. In your opinion, would merge be a better option than flatten in our case?

8

Both seem equally valid to me! Whichever ends up being clearer is the one I’d pick.

1 Like
9

Thanks for the help! We’ll try out flatten and report back if one is clearly better than the other.

10

Hi all! I’m the one that wrote that Stack Overflow answer, so I guess I should explain myself!

I typically use flatten with lists when I write examples of this pattern because the flatten function can collapse potentially many levels of nested lists at once and so it generalizes to more than two levels of nesting. The merge function doesn’t recursively collapse nested mappings, and so it can only work with a two-level heirarchy, so if you want to learn only one pattern then flatten can potentially serve more use-cases.

However, @alisdair’s example here ends up being more concise by combining the key-building into the same expression as the flattening, which is possible because there is only one additional level of nesting here.

With that said, I agree with @alisdair that both examples seem fine and I’d suggest you try both and see which one feels like a clearer representation of your intended goal. Sometime’s it’s worth writing things out a slightly longer way so that it’s easier for a future reader to understand what’s going on, but sometimes additional steps are just distracting noise; there often isn’t a single “best” answer for all situations.

1 Like
11

Hey @apparentlymart, thanks for the follow up. We’re struggling a bit with this again and are hoping you’d be able to point us in the right direction.

Here is the variable we are passing in to Terraform:

storage_accounts = {  
  1 = {               
    containers = {   
      container1a = {
        name = "container1a"
      },
      container1b = {
        name = "container1b"
      }
    }
    tier = "Standard",
    kind = "StorageV2"
  },
  2 = {
    containers = {
      container2a = {
        name = "container2a"
      },
      container2b = {
        name = "container2b"
      }
    }
    tier = "Premium",
    kind = "FileStorage"
  }
}

Here is the flattened variable we are trying to iterate through.

containers = flatten([

    for storage_account_key, storage_account in var.storage_accounts : [
      for container in storage_account.containers : {
        storage_account_key       = storage_account_key
        container_key             = container.name
        storage_account_tier      = storage_account.tier
        storage_account_kind      = storage_account.kind
      }
    ]
  ])

We are trying to create some storage containers below. The issue we are running into errors trying to retrieve the name of the storage account and the container name.

resource "azurerm_storage_container" "storagecontainer" {

  for_each = {
    for ct in local.containers : "${ct.container_key}" => ct...
  }

  name                  = "${var.sz_application}${var.sz_environment}${local.region-code}${each.value.name}"
  storage_account_name  = "${var.sz_application}${var.sz_environment}${local.region-code}sa${each.value.storage_account_key}"
  container_access_type = "private"
}

The ${each.value.name} for the name value and the ${each.value.storage_account_key} for the storage_account_name are what we’re trying to get. They should be the 1 and 2, and container1a, container1b, container2a, and container2b respectively. We’ve tried many different each.value combinations only to run into errors like so:

Error: Unsupported attribute

on .terraform\modules\tf-az-storage-account\main.tf line 67, in resource “azurerm_storage_container” “storagecontainer”:
67: storage_account_name = ${var.sz_application}${var.sz_environment}${local.region-code}sa${each.value.storage_account_key}"
|----------------
| each.value is tuple with 1 element

This value does not have any attributes.

12

Hello! Returning to this post to update with our latest findings and solution…

We were being thrown off by this error:


  on main.tf line 63, in resource "azurerm_storage_container" "storagecontainer":
  62:
  63:     for ct in local.containers : "${ct.container_key}" => ct
  64:
    |----------------
    | ct.container_key is "a"

Two different items produced the key "a" in this 'for' expression. If
duplicates are expected, use the ellipsis (...) after the value expression to
enable grouping by key.

Which resulted when we had 2 similarly named containers (even though they were in different Storage Accounts), as in this map:

storage_accounts = {
  1 = {
    containers = {
      a = {
        name = "a"
      },
      b = {
        name = "b"
      }
    }
  },
  2 = {
    containers = {
      a = {
        name = "a"
      },
      b = {
        name = "b"
      }
    }
  }
}

While containers in different Storage Accounts do not have to be unique (like the Storage Accounts do), the for_each cannot loop through duplicates. So, due to our flatten, this loop was trying to iterate through a,b,a,b. The error message made us think the ellipses would solve all of our problems. It did not.

What worked was when we changed to for_each to be as such:

resource "azurerm_storage_container" "storagecontainer" {
  for_each = {
    for ct in local.containers : "${ct.storage_account_key}${ct.container_key}" => ct
  }

  name                  = "${var.sz_application}${var.sz_environment}${local.region-code}${each.value.storage_account_key}${each.value.container_key}"
  storage_account_name  = azurerm_storage_account.storageaccount[each.value.storage_account_key].name
  container_access_type = "private"
}

In this, we are creating unique sets for our loop (in this case, 1a, 1b, 2a, 2b). We can still name our containers as needed in their own storage accounts.