Need help with nested type constraints in variable

Terraform
1

Hello. I am trying to implement multiple virtual machines per workload using a variable with nested types. I have not been able to grasp how to identify the best type constraints nor how to structure them and then call them. The variables in previous versions of the code had this split into multiple variables. Now that I am building this into a module it seems more efficient to use the workload name as the driver for resource group names, subnet names, host names, tags, etc.

I am currently using the variable below in my variables.tf

variable "workloads-test" {
  type = map(object({
    name = string
    tags = map(any)
    subnet-address-prefix = list(string)    
    vms = map(object({
      hostname  = string
      ipaddress = string
    }))
  }))
  default = {
  }
}

and populating the data below in my tfvars file

workloads-test = {
  workload1 = {
    name = "addds"
    tags = {
      Function        = "ADDS"
    }
    subnet-address-prefix = ["10.x.x.x/27"]
    vms = {
      vm1 = {
        hostname  = "ADDSBOX01"
        ipaddress = "10.x.x.x"
      }
      vm2 = {
        hostname  = "ADDSBOX02"
        ipaddress = "10.x.x.x"
      }
    }
  }
  workload2 = {
    name = "adccs"
    tags = {
      Function        = "ADCS"
    }
    subnet-address-prefix = ["10.x.x.x/27"]
    vms = {
      vm1 = {
        hostname  = "ADCSBOX01"
        ipaddress = "10.x.x.x"
      }
      vm2 = {
        hostname  = "ADCSBOX02"
        ipaddress = "10.x.x.x"
      }
    }
  }
}

The resource block errors out at the name of the NIC.

│   10:   name                          = "${each.value.vms["hostname"]}-nic"
│     ├────────────────
│     │ each.value.vms is map of object with 2 elements
│
│ The given key does not identify an element in this collection value.

I have a feeling that I am doing 1 or more or all of the following things wrong:

  1. Incorrect Type Constraints
  2. Incorrect Type Constraint variable structure
  3. Not calling on the elements properly. I am getting confused between elements and attributes and how to call them. (. vs [“”]

Any and all help is greatly appreciated.

2

Hi @Joseph2290w,

This specific error is about the reference to the value rather than about the type constraint.

You didn’t share the configuration for whatever resource this error emerged from so I’m guessing a bit, but I assume this is a resource that has for_each = var.workloads-test and so each.value here is one of the objects from that map.

Notice that the error message is saying that each.value.vms is a map of object, rather than just an object. Therefore when you write each.value.vms["hostname"] Terraform is looking for an element whose key is “hostname”, but the keys in your map are “vm1” and “vm2”, so that fails.

To make this work I think you’ll need to somehow contend with the fact that there are multiple VMs per workload, but I can’t make a specific suggestion since I don’t know what resource types we’re talking about and what name represents in this context.

If you can show more of the code you’ve already written and the rest of the error message that you truncated then I may be able to say something more specific. :grinning:

Otherwise, the only feedback I have on the constraint itself is that you should typically avoid using any except in some very rare circumstances, because it forces Terraform to guess what you intended and so it will tend to give worse feedback when a module isn’t used correctly. It seems like your tags is a map(string) and so I would suggest specifying it exactly rather than leaving Terraform to infer it. The automatic inference is working in this simple case but if a user of your module populated that attribute incorrectly Terraform is likely to generate a less helpful error message about it.

3

Thank you Sir!

I apologize ahead of time as I am not trying to make this confusing on purpose. I really really appreciate your help on this.

I am trying to create multiple Resource Groups, Subnets, NIC’s, VM’s with Tags using values from the workload-test variable based on workloads (i.e. Active Directory, Certificate Services).

I am able to create the resource groups because I am only calling attribute values from the 1st level.

This works :slight_smile:

#####------[Create Workload RG's]------#####
resource "azurerm_resource_group" "workloads-test" {
  provider = azurerm.Target-Subcription
  for_each = var.workloads-test
  name     = replace("${local.name-prefix-landing-zone}-rg", "${var.landing-zone}", "${each.value.workload}")
  location = var.location
  tags     = merge(local.calculated-tags, local.function-workload-rg, each.value.tags)
}

This does not :slightly_frowning_face:

####-----[NIC]-----####
resource "azurerm_network_interface" "workload-test" {
  provider                      = azurerm.Target-Subcription
  for_each                      = var.workloads-test
  name                          = "${each.value.vms["hostname"]}-nic"
  resource_group_name           = one([for item in azurerm_resource_group.workloads-test : item.name if can(regex("${each.value.workload}", item.name))])
  location                      = one([for item in azurerm_resource_group.workloads-test : item.location if can(regex("${each.value.workload}", item.name))])
  enable_accelerated_networking = true
  tags                          = merge(each.value.tags, local.calculated-tags, local.function-vm-nic)

  ip_configuration {
    name = "internal"
    #subnet_id                     = var.vm-subnet-id
    subnet_id                     = one([for item in azurerm_subnet.workload : item.name if can(regex("${each.value.name}", item.id))])
    private_ip_address_allocation = "Static"
    private_ip_address            = each.value.vms.ipaddress
  }
}

Here is the error:

│ Error: Invalid index
│
│   on ..\modules\compute-landing-zone\c8-virtual-machines-workload.tf line 10, in resource "azurerm_network_interface" "workload-test":
│   10:   name                          = "${each.value.vms["hostname"]}-nic"
│     ├────────────────
│     │ each.value.vms is map of object with 2 elements
│
│ The given key does not identify an element in this collection value.
╵
╷
│ Error: Invalid index
│
│   on ..\modules\compute-landing-zone\c8-virtual-machines-workload.tf line 10, in resource "azurerm_network_interface" "workload-test":
│   10:   name                          = "${each.value.vms["hostname"]}-nic"
│     ├────────────────
│     │ each.value.vms is map of object with 2 elements
│
│ The given key does not identify an element in this collection value.
╵
╷
│ Error: Missing map element
│
│   on ..\modules\compute-landing-zone\c8-virtual-machines-workload.tf line 21, in resource "azurerm_network_interface" "workload-test":
│   21:     private_ip_address            = each.value.vms.ipaddress
│     ├────────────────
│     │ each.value.vms is map of object with 2 elements
│
│ This map does not have an element with the key "ipaddress".
╵
╷
│ Error: Missing map element
│
│   on ..\modules\compute-landing-zone\c8-virtual-machines-workload.tf line 21, in resource "azurerm_network_interface" "workload-test":
│   21:     private_ip_address            = each.value.vms.ipaddress
│     ├────────────────
│     │ each.value.vms is map of object with 2 elements
│
│ This map does not have an element with the key "ipaddress".

I am have not been able to grab and use the values any lower than the first. In the example where I am trying to create the NIC I am trying different approaches for populating the NIC Name and Private IP Address. I have left a working Subnets build variable in use but would like to eventually merge that into the workload-test variable.

I have taken your advise on the type constraint used for the tags.

variable "workloads-test" {
  type = map(object({
    workload = string
    tags = map(string)
    subnet-address-prefix = list(string)    
    vms = map(object({
      hostname  = string
      ipaddress = string
    }))
  }))
  default = {
  }
}

I also want to note that I am providing the values for the var.workloads-test attributes in the tfvars file. I am in the process of converting this code to a module and I was not sure if I should even continue to use it or not as I already have a variable file in the root and the child.

Here is an updated view of that:

workloads-test = {
  workload1 = {
    workload = "addds"
    tags = {
      Function        = "ADDS"
    }
    subnet-address-prefix = ["10.x.x.x/27"]
    vms = {
      vm1 = {
        hostname  = "ADDSBOX01"
        ipaddress = "10.x.x.x"
      }
      vm2 = {
        hostname  = "ADDSBOX02"
        ipaddress = "10.x.x.x"
      }
    }
  }
  workload2 = {
    workload = "adccs"
    tags = {
      Function        = "ADCS"
    }
    subnet-address-prefix = ["10.x.x.x/27"]
    vms = {
      vm1 = {
        hostname  = "ADCSBOX01"
        ipaddress = "10.x.x.x"
      }
      vm2 = {
        hostname  = "ADCSBOX02"
        ipaddress = "10.x.x.x"
      }
    }
  }
}
4

Thanks for that additional context!

The most important requirement with for_each is that the map you provide must have one element per object you are intending to declare.

I think the main missing piece in your current attempt is that you have a collection with one element per workload but you don’t have one with an element for each distinct VM, and so you don’t have a suitable value to use to declare objects that need to exist on a per-VM basis rather than on a per-workload basis, like your NICs.

I think you can get there with a local value whose expression essentially inverts the orientation of your input variable so that the top-level elements represent VMs and each VM has one workload, rather than the other way around. Something like this:

locals {
  workload_vms = merge([
    for wlk, wl in var.workloads_test : tomap({
      for vmk, vm in wl.vms : "${wlk}:${vmk}" => {
        hostname = vm.hostname
        ip_address = vm.ip_address
        workload = wl
      }
    })
  ]...)
}

This is using two nested for expressions to transform both levels of nesting at once. The two for expressions together produce a tuple of maps of objects and then the merge function around them merges all of those maps into one bigger map where each element is a single VM.

Because your VM keys are not unique across all workloads, this new map has compound keys like "workload1:vm1" to ensure that each VM will have a unique tracking key.

I’d suggest first adding just this local value and exporting it as an output value so you can first fix any typos I’ve inevitably made as a result of typing code directly into a forum post :upside_down_face: and then also get familiar with how that data structure is shaped before you try to use it elsewhere in your module.

Once you have it working and can see how the structure is shaped, you should be able to use local.workload_vms in the for_each of any resource that needs one instance per virtual machine, which will then mean each.value is an object representing the VM and each.value.workload is the object representing whichever workload it belongs to.

(You can remove the output value once you’ve got it working; it isn’t crucial to the final implementation but just useful to see what you are working with during development.)

5

Fantastic! You Sir are a wizard. This gets me almost there. Thank you so much!!! I am going to have to read this multiple times to understand it better. I do now have a better idea on how inverting the variable with nested config helped us access the data for the VM’s. I ended up bringing more of the data to the top in the locals variable that way so I could avoid the duplication of data in the results. For example each ${wlk}:${vmk} contained both VM’s with config in the “workload” attribute below.

This below is working for me well with only one issue left I am guessing.

locals {
  workload_vms = merge([
    for wlk, wl in var.workloads_test : tomap({
      for vmk, vm in wl.vms : "${wlk}:${vmk}" => {
        hostname = vm.hostname
        ipaddress = vm.ipaddress
        #workload = wl
        workload-name = wl.name
        workload-tags = wl.tags
        workload-subnet-address-prefix = wl.subnet-address-prefix      
      }
    })
  ]...)
}

When I need to grab the subnet prefixes per workload I can continue to use the var.workload-test and when I need to use the VM information by workload I can use the local.workload_vms.

One issue I am running into is I have to use the same variable in the for_each of certain resources. For example when creating the NIC using the local.workload_vms which works great. However when I go to create a ASG I have to use var.workload-test or I end up with duplicate ASG based on there being multiple VM’s per workload.

So far that works however when I go to associate the ASG to the NIC it does not like the structure of how the ASG was created using the var.workloads-test.

Here is the error that I am getting:

Error: Invalid index
│
│   on ..\modules\compute-landing-zone\c8-virtual-machines-workload.tf line 42, in resource "azurerm_network_interface_application_security_group_association" "workload":
│   42:   application_security_group_id = azurerm_application_security_group.workload[each.key].id
│     ├────────────────
│     │ azurerm_application_security_group.workload is object with 2 attributes
│     │ each.key is "workload1:vm1"
│
│ The given key does not identify an element in this collection value.
╵
╷
│ Error: Invalid index
│
│   on ..\modules\compute-landing-zone\c8-virtual-machines-workload.tf line 42, in resource "azurerm_network_interface_application_security_group_association" "workload":
│   42:   application_security_group_id = azurerm_application_security_group.workload[each.key].id
│     ├────────────────
│     │ azurerm_application_security_group.workload is object with 2 attributes
│     │ each.key is "workload2:vm1"
│
│ The given key does not identify an element in this collection value.
╵
╷
│ Error: Invalid index
│
│   on ..\modules\compute-landing-zone\c8-virtual-machines-workload.tf line 42, in resource "azurerm_network_interface_application_security_group_association" "workload":
│   42:   application_security_group_id = azurerm_application_security_group.workload[each.key].id
│     ├────────────────
│     │ azurerm_application_security_group.workload is object with 2 attributes
│     │ each.key is "workload2:vm2"
│
│ The given key does not identify an element in this collection value.
╵
╷
│ Error: Invalid index
│
│   on ..\modules\compute-landing-zone\c8-virtual-machines-workload.tf line 42, in resource "azurerm_network_interface_application_security_group_association" "workload":
│   42:   application_security_group_id = azurerm_application_security_group.workload[each.key].id
│     ├────────────────
│     │ azurerm_application_security_group.workload is object with 2 attributes
│     │ each.key is "workload1:vm2"
│
│ The given key does not identify an element in this collection value.

Here is the code produces the error:

####-----[NIC]-----####
resource "azurerm_network_interface" "workload" {
  provider                      = azurerm.Target-Subcription
  for_each                      = local.workload_vms
  name                          = "${each.value.hostname}-nic"
  resource_group_name           = one([for item in azurerm_resource_group.workload : item.name if can(regex("${each.value.workload-name}", item.name))])
  location                      = one([for item in azurerm_resource_group.workload : item.location if can(regex("${each.value.workload-name}", item.name))])
  enable_accelerated_networking = true
  tags                          = merge(each.value.workload-tags, local.calculated-tags, local.function-vm-nic)

  ip_configuration {
    name = "internal"
    #subnet_id                     = var.vm-subnet-id
    subnet_id                     = one([for item in azurerm_subnet.workload : item.name if can(regex("${each.value.name}", item.id))])
    private_ip_address_allocation = "Static"
    private_ip_address            = each.value.ipaddress
  }
}

####-----[ASG]-----####
resource "azurerm_application_security_group" "workload" {
  provider            = azurerm.Target-Subcription
  for_each            = var.workloads-test
  name                = replace("${local.name-prefix-landing-zone}-asg", "${var.landing-zone}", "${each.value.workload-name}")
  resource_group_name = one([for item in azurerm_resource_group.workload : item.name if can(regex("${each.value.workload-name}", item.name))])
  location            = one([for item in azurerm_resource_group.workload : item.location if can(regex("${each.value.workload-name}", item.name))])
  tags                = merge(each.value.workload-tags, local.calculated-tags, local.function-asg-nic)
}

####-----[ASG Associations]-----####
resource "azurerm_network_interface_application_security_group_association" "workload" {
  provider                      = azurerm.Target-Subcription
  for_each                      = local.workload_vms
  network_interface_id          = azurerm_network_interface.workload[each.key].id
  application_security_group_id = azurerm_application_security_group.workload[each.key].id
}

Error when running plan:
is there a way to use var.workloads-test when creating the 1 ASG per workload and during the ASG Association drill down to the appropriate “application_security_group_id”

or

No error when running plan but duplicate ASG’s per Workload:
When I use local.workload_vms** in the for_each for the NIC’s, ASG’s and ASG associations the error goes away but I can see the ASG for each workload is being created multiple times (once for each VM). I am only running this in plan so I am guessing it would fail during apply.

When creating the ASG’s is there a way to scope the for_each to only 1 instance of each workload when looping through the VM’s in local.workload_vms so duplicate ASGs are not created per workload?

I am very grateful for your help and intend on paying it forward once I learn more. Do you know of any documentation out there that can help me learn the map structure for what the elements, keys and attributes are that we have been discussing when using a for_each? I have been looking but have not been able to find anything other than can break it down for me. For example the updated view I sent. Is Workload1 an element? workload is an attribute or a key? My some of these change based on type constraints. Either way the error at the moment is more important that my understanding of the structure. I am sure some of this will clear up as this is worked through.

6

I think the way I would probably try to solve this final problem is to include the workload key into the local.workload_vms structure too:

locals {
  workload_vms = merge([
    for wlk, wl in var.workloads_test : tomap({
      for vmk, vm in wl.vms : "${wlk}:${vmk}" => {
        hostname = vm.hostname
        ipaddress = vm.ipaddress
        workload_key = wlk
        workload-name = wl.name
        workload-tags = wl.tags
        workload-subnet-address-prefix = wl.subnet-address-prefix      
      }
    })
  ]...)
}

(Note that although Terraform does allow dashes as part of identifiers that’s primarily for working with remote APIs where that naming scheme is conventional; for Terraform-only identifiers it’s conventional to use underscores instead.)

Then whenever you are in a resource which is using for_each = local.workload_vms you can access the corresponding workload key as each.value.workload_key. For example

application_security_group_id = azurerm_application_security_group.workload[each.value.workload_key].id

This gives you a route back across the many-to-one relationship between the VMs and the workloads, so that all of the VMs for the same workload can all refer to the same ID.

7

Thank you Sir. This effort was extremely enlightening! My original request for assistance I thought was type constraint related when it was really more about how to access multi layered data during the loop. I wish I could rename the title so others might be able to find the solution that matches the titled issue. Perhaps something like “Accessing multi level structure for_each data during deployment” Maybe that statement alone would help others find this article :slight_smile:

I still have a ways to go with understanding it all but I was able to replicate the inverted mapping work to include something for an lb, front end ip and backend vm’s. Now I can deploy resource groups, subnets, route tables, nic’s, asg’s, nsgs, vm’s with load balancer, probe, rules back end pool and pool association all with the appropriate naming and tags PER WORKLOAD! using the same variable. The variable now looks like this. I did try to add in Disk Size using number as the type constraint with no success but that is a battle for another day.

#####------[Workloads]------#####
variable "workloads" {
  type = map(object({
    workload-name                  = string
    workload-tags                  = map(string)
    workload-subnet-address-prefix = list(string)
    vms = map(object({
      hostname  = string
      ipaddress = string
      vm-size   = string
    }))
    lb-frontend-private-ip-address = string
    lb-backend-vms                  = map(object({
      hostname  = string
    }))
  }))
  default = {
  }
}

I even added filters on the for_each statements to help scope resource creation.

###----[Create Web Rule - 443]----###
resource "azurerm_lb_rule" "https-workload" {
  provider                       = azurerm.Target-Subcription
  for_each                       = { for i, item in var.workloads : i => item if item.lb-frontend-private-ip-address != "" }
  loadbalancer_id                = azurerm_lb.workload[each.key].id
  backend_address_pool_ids       = [azurerm_lb_backend_address_pool.workload[each.key].id]
  probe_id                       = azurerm_lb_probe.https-workload[each.key].id
  disable_outbound_snat          = true
  name                           = "bepool_web_rule_https"
  protocol                       = "Tcp"
  frontend_port                  = 443
  backend_port                   = 443
  frontend_ip_configuration_name = azurerm_lb.workload[each.key].frontend_ip_configuration[0].name
}

I appreciate the syntax advise on the identifiers “-” vs “_” and will adjust appropriately for consistency.

All that being said I think I am good to go. Thank you so very much for your time Sir.