Using for_each with templatefile and input module outputs

I have a module that creates one instance, one proxy instance, two external IPs, and firewall rules for the instances. To invoke the module I using for_each to loop through the module and create the desired number of instances and proxies for a ephemiral environment. Below is how this is constructed

module.tf

#--------Node--------#

resource "google_compute_address" "ext1" {
  name         = "${var.deployment_name}-ext1"
  address_type = "EXTERNAL"
  # Trim the trailing zone identifier to get the region from the existing variables
  region  = tostring(join("", regex("^(.*)-[a-z]$", var.node_az)))
  project = var.project
}


resource "google_compute_address" "ext2" {
  name         = "${var.deployment_name}-ext2"
  address_type = "EXTERNAL"
  # Trim the trailing zone identifier to get the region from the existing variables
  region  = tostring(join("", regex("^(.*)-[a-z]$", var.node_az)))
  project = var.project
}

resource "google_compute_instance" "node" {
  name         = var.deployment_name
  machine_type = var.node_machine_type
  zone         = var.node_az
  project      = var.project
  .....<truncated>.....
}

resource "google_compute_firewall" "node" {
.....<truncated>.....
}

#--------Proxy--------#

resource "google_compute_address" "proxy" {
  name         = "${var.deployment_name}-proxy-ext"
  address_type = "EXTERNAL"
  # Trim the trailing zone identifier to get the region from the existing variables
  region  = tostring(join("", regex("^(.*)-[a-z]$", var.proxy_az)))
  project = var.project
}

resource "google_compute_instance" "proxy" {
  name         = "${var.deployment_name}-proxy"
  machine_type = var.proxy_machine_type
  zone         = var.proxy_az
  project      = var.project
.....<truncated>.....
}

resource "google_compute_firewall" "proxy" {
.....<truncated>.....
}

#--------Outputs--------#
output "node_instance_id" {
  value = google_compute_instance.node.instance_id
}

output "node_internal_instance_ip" {
  value = google_compute_instance.node.network_interface[0].network_ip
}

output "node_external_instance_ip" {
  value = google_compute_instance.node.network_interface[0].access_config[0].nat_ip
}

// external_instance_ip and external_ip_1 should be the same
output "node_external_ip1" {
  value = google_compute_address.ext1.address
}

output "gcp_proxy_external_ip" {
  value = google_compute_address.proxy.address
}

output "shard" {
  value = replace(var.shard, "-", ".")
}

output "conf_id" {
  value = var.conf_id
}

output "node_info" {
  value = tomap({
    conf_id      = var.conf_id
    shard        = var.shard
    internal_ip  = google_compute_instance.node.network_interface[0].network_ip
    external_ip1 = google_compute_address.ext1.address
  })
}

This module is called using for_each:

module "nodes-and-proxies" {
  source          = "git@github.com:rustyShacklefurd/terraform-instance-proxy-module.git"
  for_each        = var.node_data
  deployment_name = "${terraform.workspace}-${each.key}"
  proxy_disk_size    = local.proxy_disk_size
  node_disk_size     = local.node_disk_size
  project            = var.project
  node_id            = each.key
  node_az            = each.value["node_az"]
  proxy_az           = each.value["proxy_az"]
  shard              = each.value["shard"]
  conf_id = each.value["conf_id"]
}

-------Vars--------
node_data = {
  node1 = {
    node_az  = "us-east1-b"
    proxy_az = "us-east1-d"
    shard    = "0-0-3"
    conf_id  = "A"
  },
  node2 = {
    node_az  = "us-east4-c"
    proxy_az = "us-east4-b"
    shard    = "0-0-4"
    conf_id  = "B"
  },
  node3 = {
    node_az  = "us-central1-a"
    proxy_az = "us-central1-b"
    shard    = "0-0-5"
    conf_id  = "C"
  },
  node4 = {
    node_az  = "us-west1-b"
    proxy_az = "us-west1-c"
    shard    = "0-0-6"
    conf_id  = "D"
  },
  node5 = {
    node_az  = "us-west2-a"
    proxy_az = "us-west2-b"
    shard    = "0-0-7"
    conf_id  = "E"
  },
  node6 = {
    node_az  = "us-west3-a"
    proxy_az = "us-west3-b"
    shard    = "0-0-8"
    conf_id  = "F"
  },
  node7 = {
    node_az  = "us-central1-f"
    proxy_az = "us-central1-c"
    shard    = "0-0-9"
    conf_id  = "G"
  }
}

The creation of the IPs, instance, proxy, and firewall work as expected here. However, the next step is use the outputs from invoking the module to generate a yaml file for ansible. I have been having difficulties in getting for_each to work with templatefile calls. After sometime trying different approaches I have found a way that does not error out but it doesn’t create the yaml file as expected either.
Here is the templatefile call and template
templatefile resource:

resource "local_file" "ansible_hosts_nodes" {
  for_each = module.nodes-and-proxies
  content = templatefile("templates/hosts.tpl", {
    hostname = each.key
    node_external_ip = each.value.node_external_ip1
    shard = each.value.shard
    node_num = replace (each.key, "node", "")
  })
  depends_on = [
    module.nodes-and-proxies
  ]  
  filename = "test-hosts.txt"
}

hosts.tpl template:

    ${hostname}:
      ansible_ssh_host: ${node_external_ip}
      NODE_ID: ${shard}
      NODE_NUM: ${node_num}

When this is run the for_each on the module creates all the resources correctly but the output from the templatefile is always one instance information being populated instead of many instances being populated.
example test-host.txt output:

    node2:
      ansible_ssh_host: 35.245.219.226
      NODE_ID: 0.0.4
      NODE_NUM: 2

I suspect there needs to be a loop instead of the template but it is unclear to me how to get the template loop and for_each setup to work together to achieve this. Any ideas or things to test would be greatly appreciated. Thank you.

Did you ever get this working?

I didn’t see this the first time around but the OP was on the right track about this being a matter of what exactly is being repeated.

Each local_file instance represents a separate file. For most resource types a provider will detect if it’s being asked to create something that already exists and return an error, but local_file in particular seems to just overwrite an existing file and so on effect all of the instances of this resource are racing to write to the same filename test-hosts.txt. Using for_each this way would require each instance to have a separate filename so that they can all coexist.

However, the goal here was to have one file describing multiple objects. One file means one instance of local_file, and so this is not an appropriate situation for resource for_each.

Instead, the template itself needs to see the full list of objects all as one input, which it can then use to generate repeated content using the for template directive.

The first step here would be to move the projection of module.nodes-and-proxies down into the templatefile argument so that the single template call can see all of the entries at once:

  content = templatefile("templates/hosts.tpl", {
    nodes = tomap({
      for hostname, node in module.nodes-and-proxies : k => {
        hostname = hostname
        node_external_ip = node.node_external_ip1
        shard = node.shard
        node_num = replace(hostname, "node", "")
      }
    })
  })

The template scope now has just a single variable nodes that is a map of objects. You can use the template for directive to produce a sequence of repeated chunks based on the contents of this map.

This particular situation seems to be generating YAML and so template interpolation and directives would not actually be an appropriate answer here. Instead, I would suggest doing what’s described in Generating JSON or YAML from a template, because that will guarantee that the result is always valid YAML without any weird workarounds to deal with individuals quoting and escaping problems.