Outputs from child module which is created by for_each

Hi guys,

I’m hoping you can help, we recently upgraded our version of Terraform from version 0.12.26 to 0.13.4… Since then we have started to use for_each on our root modules. Now that we have done this, we can no longer use the outputs generated by the child module in our root main.tf like we could previously.

More specifically, we have root modules in our root-level main.tf file which call child modules to create either a Linux or Windows VM. This all works well, VM’s, NIC’s, NSG’s and disks are created as expected. However, when we then try to read the output from said child module back into the root module to then add NSG rules to the relevant NSG’s it fails with:

Error: Incorrect attribute value type

on main.tf line 69, in resource “azurerm_network_security_rule” “fico-app-sr-80”:
69: network_security_group_name = module.fico_app_vm.linux_vm_nsg

Inappropriate value for attribute “network_security_group_name”: string
required.

If I then update that line to add double quotes (network_security_group_name = “module.fico_app_vm.linux_vm_nsg”) to get around the type error, we see the following error:

Error: Error Creating/Updating Network Security Rule “nsr-sbox-dd-fed-ddsp-http80” (NSG “module.fico_app_vm.linux_vm_nsg” / Resource Group “rg-dwp-sbox-dd-fed-ddsp-app”): network.SecurityRulesClient#CreateOrUpdate: Failure sending request: StatusCode=404 – Original Error: Code=“ResourceNotFound” Message="The Resource ‘Microsoft.Network/networkSecurityGroups/module.fico_app_vm.linux_vm_nsg’ under resource group ‘rg-dwp-sbox-dd-fed-ddsp-app’ was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"

on main.tf line 58, in resource “azurerm_network_security_rule” “fico-app-sr-80”:

58: resource “azurerm_network_security_rule” “fico-app-sr-80” {

I have checked in the Azure Console that the NSG has been created in the correct resource group, I believe the module output isn’t being populated, probably passing it as a literal string due to the double quotes I used to get around the type error.

I have included my tf files below to help debug. Please let me know if there is any other information I should provide.

> root module .tfvars:

**
app_servers ={
app-1 = {
size = “Standard_E2s_v3”
admin_username = “azureuser”
public_key = “XXX”
disks = [32, 32]
zone_vm = “1”
zone_disk = [“1”]
}
}

root module main.tf:

module "fico_app_vm" {
  for_each                     = var.app_servers
  source                       = "../modules/compute/linux_vm"
  source_image_id              = var.app_image_id
  location                     = var.location
  vm_name                      = each.key
  vm_identifier                = "${var.vm_identifier}${var.instance_number}"
  vm                           = each.value
  disks                        = each.value["disks"]
  resource_group               = azurerm_resource_group.rg_dwp_fico_app.name
  directorate                  = var.directorate
  business_unit                = var.business_unit
  environment                  = var.environment
  network_rg_identifier        = var.network_rg_identifier
  subnet_name                  = "sub-dwp-${var.environment}-${var.directorate}-${var.business_unit}-be01"
  diag_storage_account_name    = var.diag_storage_account_name
  ansible_storage_account_name = var.ansible_storage_account_name
  ansible_storage_account_key  = var.ansible_storage_account_key
  log_analytics_workspace_name = var.log_analytics_workspace_name
  backup_policy_name           = var.backup_policy_name
  enable_management_locks      = true
}

root level variables.tf:

variable “app_servers” {
description = “Variable for defining each instance”
type = map(object({
size = string
admin_username = string
public_key = string
disks = list(number)
zone_vm = string
zone_disk = list(string)
}))
}

child module main.tf:

resource “azurerm_network_security_group” “dwp_network_security_group” {
name = “nsg-{var.environment}-{var.directorate}-{var.business_unit}-{var.vm_identifier}-${var.vm_name}”
resource_group_name = var.resource_group
location = var.location
}

child module outputs.tf:

output “linux_vm_nsg” {
value = azurerm_network_security_group.dwp_network_security_group.name
}

root module main.tf (trying to reference child module output to set NSG rules):

resource “azurerm_network_security_rule” “fico-app-sr-80” {
name = “nsr-{var.environment}-{var.directorate}-{var.business_unit}-{var.vm_identifier}${var.instance_number}-http80”
priority = 100
direction = “Inbound”
access = “Allow”
protocol = “"
source_port_range = "

destination_port_range = “80”
source_address_prefixes = [“module.fico_web_vm.linux_vm_ips”]
destination_address_prefix = “VirtualNetwork”
resource_group_name = “azurerm_resource_group.rg_dwp_fico_app.name”
network_security_group_name = module.fico_app_vm.linux_vm_nsg
}

1 Like

Apologies for the poor formatting in certain areas (itallics) - I have tried to correct it but ended up messing it up further in the preview!

So you have added a for_each to the module fico_app_vm in main.tf that you didn’t have before?

If so, instead of having a single instance of the module you now have multiple instances (depending on the content of var.app_servers). As a result (and similar to using count) you have to reference a specific instance using the method.

So for example module.fico_app_vm[“xxx”].linux_vm_nsg

You would generally be referencing that from a resource which itself is in a for_each or where you know what the “xxx” should be. If in a for_each you might use each.key/each.value

1 Like

Thanks for the response @stuart-c - I will give this a try by using:

module.fico_app_vm[each.key].linux_vm_nsg

@stuart-c - I can confirm that adding [each.key] like below works:

module.fico_app_vm[each.key].linux_vm_nsg

However, the problem we’re now faced with is attempting to use this method for a resource which requires the output from two different modules. For example, we’re trying to add an NSG rule for our DB server. We require an output from the child module which creates the DB as well as the output from the child module which creates the APP server to use as the source address (allowing inbound connections from the APP server). See below:

resource "azurerm_network_security_rule" "fico-db-sr-1433" {
  for_each                     = var.db_servers
  name                        = "nsr-${var.environment}-${var.directorate}-${var.business_unit}-${var.vm_identifier}${var.instance_number}-sql"
  priority                    = 100
  direction                   = "Inbound"
  access                      = "Allow"
  protocol                    = "*"
  source_port_range           = "*"
  destination_port_range      = "1433"
  source_address_prefixes     = module.fico_app_vm[each.key].linux_vm_ips
  destination_address_prefix  = "VirtualNetwork"
  resource_group_name         = azurerm_resource_group.rg_dwp_fico_db.name
  network_security_group_name = module.fico_db_vm[each.key].windows_vm_nsg
}

With this in place, I’m now presented with the following error:

Error: Invalid index

  on main.tf line 345, in resource "azurerm_network_security_rule" "fico-db-sr-1433":
 345:   source_address_prefixes     = module.fico_app_vm[each.key].linux_vm_ips
    |----------------
    | each.key is "db-1"
    | module.fico_app_vm is object with 1 attribute "app-1"

The given key does not identify an element in this collection value.

I have tried to get around this by creating an outputs.tf in the root module and then referencing it in the resource as well as defining it as a local variable. Of course neither work as each.key can only be specified in modules/resources directly… I’m a bit stumped.

Is it possible to do this now with for_each?

Yes, you just need to pick the right set to loop over in your for_each. In this case as you are wanting to have a resource for each database server, for each app server you need to loop over something which is a Cartesian product of the two (for example with 2 database servers and 3 app servers you want 6 resources).

The setproduct() function should be useful

Something similar to:

resource "azurerm_network_security_rule" "fico-db-sr-1433" {
    for_each                    = setproduct(["db1", "db2"], ["app1", "app2", "app3"])
    source_address_prefixes     = module.fico_app_vm[each.key[1]].linux_vm_ips
    network_security_group_name = module.fico_db_vm[each.key[0]].windows_vm_nsg
    ...
}

It looks like you could get the list of app servers using something like keys(var.app_servers).

Thanks for the reply again @stuart-c

I can see how your example with setproduct() would work, my only concern is that this isn’t dynamic - We may require varying amount of servers per environment.

When you suggest keys(var.app_servers) - Would this allow this resource to be dynamic in that sense? I’m struggling to understand how that would fit in the resource, I must admit I am a little out of my depth with this level of complexity in Terraform!

I have tried:

resource “azurerm_network_security_rule” “fico-db-sr-1433” {
for_each = setproduct(keys([var.db_servers], [var.app_servers]))
name = “nsr-{var.environment}-{var.directorate}-{var.business_unit}-{var.vm_identifier}${var.instance_number}-sql”
priority = 100
direction = “Inbound”
access = “Allow”
protocol = “"
source_port_range = "

destination_port_range = “1433”
source_address_prefixes = module.fico_app_vm[each.key[1]].linux_vm_nsg
destination_address_prefix = “VirtualNetwork”
resource_group_name = azurerm_resource_group.rg_dwp_fico_db.name
network_security_group_name = module.fico_db_vm[each.key[0]].windows_vm_nsg
}

But this returns the following errors:

Error: Too many function arguments

on main.tf line 337, in resource “azurerm_network_security_rule” “fico-db-sr-1433”:
337: for_each = setproduct(keys([var.db_servers], [var.app_servers]))

Function “keys” expects only 1 argument(s).

Error: Invalid index

on main.tf line 345, in resource “azurerm_network_security_rule” “fico-db-sr-1433”:
345: source_address_prefixes = module.fico_app_vm[each.key[1]].linux_vm_nsg

This value does not have any indices.

Error: Invalid index

on main.tf line 348, in resource “azurerm_network_security_rule” “fico-db-sr-1433”:
348: network_security_group_name = module.fico_db_vm[each.key[0]].windows_vm_nsg

This value does not have any indices.

That was just a simple example. Yes you would need to set the for_each value using something that isn’t a fixed list.

The example I gave for keys(var.app_servers) returns the map keys for that variable, which seems to be the names you are also using in the for_each for the app module.

keys() expects a single map as the parameter [keys(var.map_name)] so you need to run it twice - once for the app server variable and once for the db server variable.

So something like for_each = setproduct(keys(var.db_servers), keys(var.app_servers)))

1 Like

That makes sense, I came to the same understanding after some Googling (I should’ve updated my last question). D’oh!

However, with the above in place it’s now struggling with the index references…

Error: Invalid index

on main.tf line 345, in resource “azurerm_network_security_rule” “fico-db-sr-1433”:
345: source_address_prefixes = module.fico_app_vm[each.key[1]].linux_vm_nsg

This value does not have any indices.

Error: Invalid index

on main.tf line 348, in resource “azurerm_network_security_rule” “fico-db-sr-1433”:
348: network_security_group_name = module.fico_db_vm[each.key[0]].windows_vm_nsg

This value does not have any indices

Try wrapping the setproduct() in toset() [i.e. toset(setproduct(..., ...))

Actually it looks like you need to have a map, as the set should be strings.

See setproduct - Functions - Configuration Language - Terraform by HashiCorp

@stuart-c - Thanks for the response.

We are already setting our db_servers and app_servers as a map:

root level variables.tf

variable “db_servers” {
description = “Variable for defining each instance”
type = map(object({
size = string
admin_user = string
admin_password = string
disks = list(number)
zone_vm = string
zone_disk = list(string)
}))
}

variable “app_servers” {
description = “Variable for defining each instance”
type = map(object({
size = string
admin_username = string
public_key = string
disks = list(number)
zone_vm = string
zone_disk = list(string)
}))
}

I followed the link to the setproduct example but I’m not sure how it could help us as we don’t require the values which the example sets - We just need to be able to reference the relevant key.

e.g.

toset(setproduct(keys(var.db_servers), keys(var.app_servers)))

source_address_prefixes = module.fico_app_vm[each.key[1]].linux_vm_nsg

network_security_group_name = module.fico_db_vm[each.key[0]].windows_vm_nsg

If I wrap setproduct() in toset() like you said previously I get the same error as previously:

Error: Invalid index

on main.tf line 345, in resource “azurerm_network_security_rule” “fico-db-sr-1433”:
345: source_address_prefixes = module.fico_app_vm[each.key[1]].linux_vm_nsg

This value does not have any indices.

Error: Invalid index

on main.tf line 348, in resource “azurerm_network_security_rule” “fico-db-sr-1433”:
348: network_security_group_name = module.fico_db_vm[each.key[0]].windows_vm_nsg

This value does not have any indices.