Dereferencing a map when using template_file function

[chrisadkin]

In have the following variable:

variable "virtual_machines" {
  default = {
    "master1" = {
       name         = "z-ca-bdc-master1"
       worker_node  = false
       ipv4_address = "192.168.113.79"
       ipv4_netmask = "22"
       ipv4_gateway = "192.168.112.1"
       dns_server   = "192.168.112.2"
       ram          = 8192
       logical_cpu  = 4
       disk0_size   = 40
       disk1_size   = 0
    },
    "master2" =  {
       name         = "z-ca-bdc-master2"
       worker_node  = false
       ipv4_address = "192.168.113.80"
       ipv4_netmask = "22"
       ipv4_gateway = "192.168.112.1"
       dns_server   = "192.168.112.2"
       ram          = 8192
       logical_cpu  = 4
       disk0_size   = 40
       disk1_size   = 0
    },

And I would like to iterate through this in order to use the element worker_node and name in a template, this is the code I’m using which is clearly wrong because terraform plan spits it out:

data "template_file" "k8s" {
  template = file("./templates/kubespray_inventory.tpl")

  vars = {
    k8s_master_name = join("\n", var.virtual_machines.worker_node ? "" : var.virtual_machines.name)
    k8s_node_host   = join("\n", var.virtual_machines.name)
  }
}

Can someone tell me what the correct syntax is that I should be using in place of var.virtual_machines.worker_node and var.virtual_machines.name.

[tbugfinder]

I looks like count or for_each could help without knowing the .tpl file and resources within the dependency tree.

data "template_file" "k8s" {
  for_each = var.virtual_machines
  template = file("./templates/kubespray_inventory.tpl")

  vars = {
    k8s_master_name = join("\n", each.value.worker_node ? "" : each.value.name)
    k8s_node_host   = join("\n", each.value.name)
  }
}

[chrisadkin]

Yes it does indeed help, this is what I’ve now got:

data "template_file" "k8s" {
  for_each = var.virtual_machines
  template = file("./templates/kubespray_inventory.tpl")

  vars = {
    k8s_master_name = each.value.worker_node ? "" : join("\n", each.value.name)
    k8s_node_host   = concat(["", "\n"], each.value.name)
  }
}

However, terraform plan is giving me:

Error: Invalid function argument

on main.tf line 50, in data "template_file" "k8s":
50:     k8s_node_host   = concat(["", "\n"], each.value.name)
     |----------------
     | each.value.name is "z-ca-bdc-worker1"

Invalid value for "seqs" parameter: all arguments must be lists or tuples; got
string.

I have tried various things to work around this, permutations of cancat and join but to no avail. Thanks for your help, please advise further. All I need to be able to do is to join “\n” and each.value.name .

[tbugfinder]

@chrisadkin

each.value.name

that’s not a list, however try:

concat(["", "\n"], [each.value.name])

[chrisadkin]

Getting further, this is what terraform plan liked:

data "template_file" "k8s" {
  for_each = var.virtual_machines
  template = file("./templates/kubespray_inventory.tpl")

  vars = {
    k8s_master_host = each.value.worker_node ? "" : join("\n", [each.value.name])
    k8s_node_host   = join("\n", [each.value.name])
  }
}

The one last hurdle I need to get past is a problem with rendering the file:

resource "local_file" "k8s_file" {
  for_each = { for i, j in var.virtual_machines : j => { index = i } }
  content = data.template_file.k8s[each.value.name].rendered
  filename = "./k8s-host"
}


Error: Invalid object key

  on main.tf line 55, in resource "local_file" "k8s_file":
  55:   for_each = { for i, j in var.virtual_machines : j => { index = i } }

The key expression produced an invalid result: string required.

Many thanks for your assistance and time.

[tbugfinder]

@chrisadkin

j => { index = i }

j is a map so that’s not possible.
What do you expect here?

[apparentlymart]

Hi @chrisadkin,

Since you are using Terraform v0.12 or later, I would suggest using the built-in templatefile function instead of the template provider. It integrates better into the language and, in particular for your case, doesn’t have the restriction that all of the assigned variables must be strings.

Since templatefile is just a normal function, you can call it from any place where expressions are allowed, but for the sake of an example I’m going to declare a template result as a local value:

locals {
  k8s = {
    for key, vm in var.virtual_machines :
    key => templatefile("${path.module}/kubespray_inventory.tpl", {
      k8s_master_node = vm.worker_node ? "" : vm.name
      k8s_node_host   = vm.name
    })
  }
}

I’m not sure I fully followed what your intent was for each of the expressions in your example so I’m sorry if I didn’t get that exactly as you wanted it, but hopefully the above shows a suitable structure to start with and you can adapt it to what you need. If not, I’m happy to answer follow-up questions if you can share what result you got and how that differed from the result you were expecting.

[chrisadkin]

I managed to come up with something, not as elegant as what you have got, how do I render k8s (from your example) as an actual file ?

[chrisadkin]

In you example, how would I dereference k8s_master_node, would this be local.k8s.k8s_master_node ?

[apparentlymart]

local.k8s is an in-memory map from virtual machine key to rendered string, so local.k8s.master1 would be the result of rendering for the master1 element, for example.

Terraform configurations typically pass template results to remote network services rather than local files, because local files won’t persist from one run to the next unless you’re keeping them somewhere persistent and so your configuration will never converge on a completely-applied state, but if you do intend to write those files to somewhere persistent (like a shared filesystem) then I suppose you could do it like this:

variable "config_store_path" {
  type = string
}

resource "local_file" "example" {
  for_each = local.k8s

  filename = "${var.config_store_path}/${each.key}"
  content  = each.value
}

Please note the caveats stated in the local_file resource type’s documentation if you intend to actually use something like the above, but the same sort of pattern above would apply to any resource type where you’d want to declare one for each of the rendered results.

[chrisadkin]

This is what I have at present, it works, but its a mess:

locals {
   all_nodes_verbose_etcd = [for k, v in var.virtual_machines:
                               format("%s ip=%s etcd_instance=%s", v.name, v.ipv4_address, v.etcd_instance)
                               if length(v.etcd_instance) > 0]
   all_nodes_verbose      = [for k, v in var.virtual_machines: format("%s ip=%s", v.name, v.ipv4_address) if length(v.etcd_instance) == 0]
   master_nodes           = [for k, v in var.virtual_machines: v.name if v.worker_node != true]
   etcd_nodes             = [for k, v in var.virtual_machines: v.name if length(v.etcd_instance) > 0]
   all_nodes              = values(var.virtual_machines)[*].name
   kubespray_inv_file = format("~/kubespray/inventory/%s/inventory.ini", var.kubespray_inventory)
}

resource "local_file" "kubespray_inventory" {
  content = templatefile("templates/kubespray_inventory.tpl", {
    k8s_node_host_verbose_etcd = replace(join("\", \"\n", local.all_nodes_verbose_etcd), "\", \"", "")
    k8s_node_host_verbose      = replace(join("\", \"\n", local.all_nodes_verbose), "\", \"", "")
    k8s_master_host            = replace(join("\", \"\n", local.master_nodes), "\", \"", "")
    k8s_etcd_host              = replace(join("\", \"\n", local.etcd_nodes), "\", \"", "")
    k8s_node_host              = replace(join("\", \"\n", local.all_nodes), "\", \"", "")
  })
  filename = local.kubespray_inv_file
}

It creates a single file, which is what I want, do you have any suggestions for how I could clean this up ?

[apparentlymart]

I’m not famililar enough with “kubespray” to understand what the desired result would be here. Can you show an example of what the contents of that inventory.ini file ought to be, and then I can try to write an example of how to generate that?

[chrisadkin]

This is what the file looks like (my end goal):

[all]
z-ca-bdc-master1 ip=192.168.113.79 etcd_instance=etcd1
z-ca-bdc-master2 ip=192.168.113.80 etcd_instance=etcd2
z-ca-bdc-worker1 ip=192.168.113.81 etcd_instance=etcd3
z-ca-bdc-worker2 ip=192.168.113.82
z-ca-bdc-worker3 ip=192.168.113.83

[kube-master]
z-ca-bdc-master1
z-ca-bdc-master2

[etcd]
z-ca-bdc-master1
z-ca-bdc-master2
z-ca-bdc-worker1

[kube-node]
z-ca-bdc-master1
z-ca-bdc-master2
z-ca-bdc-worker1
z-ca-bdc-worker2
z-ca-bdc-worker3

[calico-rr]

[k8s-cluster:children]
kube-master
kube-node
calico-rr

and this is the source list of maps I use to create it from:

variable "virtual_machines" {
  default = {
    "master1" = {
       name          = "z-ca-bdc-master1"
       worker_node   = false
       etcd_instance = "etcd1"
       ipv4_address  = "192.168.113.79"
       ipv4_netmask  = "22"
       ipv4_gateway  = "192.168.112.1"
       dns_server    = "192.168.112.2"
       ram           = 8192
       logical_cpu   = 4
       disk0_size    = 40
       disk1_size    = 0
    },
    "master2" =  {
       name          = "z-ca-bdc-master2"
       worker_node   = false
       etcd_instance = "etcd2"
       ipv4_address  = "192.168.113.80"
       ipv4_netmask  = "22"
       ipv4_gateway  = "192.168.112.1"
       dns_server    = "192.168.112.2"
       ram           = 8192
       logical_cpu   = 4
       disk0_size    = 40
       disk1_size    = 0
    },
    "worker1" = {
       name          = "z-ca-bdc-worker1"
       worker_node   = true
       etcd_instance = "etcd3"
       ipv4_address  = "192.168.113.81"
       ipv4_netmask  = "22"
       ipv4_gateway  = "192.168.112.1"
       dns_server    = "192.168.112.2"
       ram           = 69632
       logical_cpu   = 12
       disk0_size    = 40
       disk1_size    = 80
    },
    "worker2" = {
       name          = "z-ca-bdc-worker2"
       worker_node   = true
       etcd_instance = ""
       ipv4_address  = "192.168.113.82"
       ipv4_netmask  = "22"
       ipv4_gateway  = "192.168.112.1"
       dns_server    = "192.168.112.2"
       ram           = 69632
       logical_cpu   = 12
       disk0_size    = 40

.
.
.

[apparentlymart]

Thanks for that extra context, @chrisadkin.

With those requirements, I think I’d prefer to pass the entire virtual machines data structure as-is into the template, and then use the template directives to implement the translation into the template syntax.

Within the main configuration then, you’d have a relatively-simple templatefile call. I’m going to show it again as a local value just because expressions need to be associated with something, but you could inline this templatefile call somewhere else in your configuration if that makes the result more readable or easier to maintain.

variable "virtual_machines" {
  type = map(object({
    name          = string
    worker_node   = bool
    etcd_instance = string
    ipv4_address  = string
    ipv4_netmask  = number
    ipv4_gateway  = string
    dns_server    = string
    ram           = number
    logical_cpu   = number
    disk0_size    = number
    disk1_size    = number
  }))
}

locals {
  kubespray_inventory = templatefile("${path.module}/templates/kubespray_inventory.tpl", {
    vms = var.virtual_machines
  })
}

Then the template itself is where all the interesting logic would be:

[all]
%{ for vm in vms ~}
${vm.name} ip=${vm.ipv4_address}%{ if vm.etcd_instance != null } etcd_instance=${vm.etcd_instance}%{ endif }
%{ endfor ~}

[kube-master]
%{ for vm in vms ~}
%{ if !vm.worker_node ~}
${vm.name}
%{ endif ~}
%{ endfor ~}

[etcd]
%{ for vm in vms ~}
%{ if vm.etcd_instance != null ~}
${vm.name}
%{ endif ~}
%{ endfor ~}

[kube-node]
%{ for vm in vms ~}
${vm.name}
%{ endfor ~}

[calico-rr]

[k8s-cluster:children]
kube-master
kube-node
calico-rr

Hopefully I understood correctly what you intended here, or at least that this example is a good starting point for tweaking it to the correct logic to match your requirements.

[chrisadkin]

Many thanks, I will take a look at this,