Transform Terraform Data Structure

Terraform
1

Dear All,

I’m seeking help for a problem I cannot seem to resolve despite many sunk hours; all likely lack of understanding.

I’m trying to transform data from a module output into a map that for_each can consume. I’ve been through posts discussing the use of flatten() but I can’t seem to make it work for my use-case. If someone can assist I’d be very grateful.

  • Source module output data
> var.test_data_map
{
  "vm_amis" = [
    "ami-067436817bb0256c1",
    "ami-067436817bb0256c1",
  ]
  "vm_availability_zones" = [
    "eu-central-1a",
    "eu-central-1b",
  ]
  "vm_hostnames" = [
    "ecas01",
    "ecas02",
  ]
  "vm_ids" = [
    "i-013d3a8efb84bfdbf",
    "i-06e869a7c822b34f6",
  ]
  "vm_instance_type" = [
    "t2.medium",
    "t2.medium",
  ]
  "vm_private_dns" = [
    "ip-10-10-10-37.eu-central-1.compute.internal",
    "ip-10-10-10-88.eu-central-1.compute.internal",
  ]
  "vm_private_ips" = [
    "10.10.10.37",
    "10.10.10.88",
  ]
  "vm_subnet_ids" = [
    "subnet-075464d83a0ce90f8",
    "subnet-0102e530cae8471ed",
  ]
}
  • From this variable 
variable "test_data_map" {
  //type = map(list(string))
  type = any
  default = {
    "vm_amis"               = ["ami-067436817bb0256c1", "ami-067436817bb0256c1"]
    "vm_availability_zones" = ["eu-central-1a", "eu-central-1b"]
    "vm_hostnames"          = ["ecas01", "ecas02"]
    "vm_ids"                = ["i-013d3a8efb84bfdbf", "i-06e869a7c822b34f6"]
    "vm_instance_type"      = ["t2.medium", "t2.medium"]
    "vm_private_dns"        = ["ip-10-10-10-37.eu-central-1.compute.internal", "ip-10-10-10-88.eu-central-1.compute.internal"]
    "vm_private_ips"        = ["10.10.10.37", "10.10.10.88"]
    "vm_subnet_ids"         = ["subnet-075464d83a0ce90f8", "subnet-0102e530cae8471ed"]
  }
}
  • If I run the following on the data it’s close to how I need the data, but not quite.
> { for k,v in var.test_data_map : k => v[0] }
{
  "vm_amis" = "ami-067436817bb0256c1"
  "vm_availability_zones" = "eu-central-1a"
  "vm_hostnames" = "ecas01"
  "vm_ids" = "i-013d3a8efb84bfdbf"
  "vm_instance_type" = "t2.medium"
  "vm_private_dns" = "ip-10-10-10-37.eu-central-1.compute.internal"
  "vm_private_ips" = "10.10.10.37"
  "vm_subnet_ids" = "subnet-075464d83a0ce90f8"
}
  • To be consumed by for_each I think I need the data structure to be a map and ideally representative of this, but can’t work out via flatten() or lambda functions how to transform it properly. Using flatten I ended up with many maps not the 2x required.
i-013d3a8efb84bfdbf" = {
  "vm_amis" = "ami-067436817bb0256c1"
  "vm_availability_zones" = "eu-central-1a"
  "vm_hostnames" = "ecas01"
  "vm_ids" = "i-013d3a8efb84bfdbf"
  "vm_instance_type" = "t2.medium"
  "vm_private_dns" = "ip-10-10-10-37.eu-central-1.compute.internal"
  "vm_private_ips" = "10.10.10.37"
  "vm_subnet_ids" = "subnet-075464d83a0ce90f8"
}
"i-06e869a7c822b34f6" = {
  "vm_amis" = "ami-067436817bb0256c1"
  "vm_availability_zones" = "eu-central-1b"
  "vm_hostnames" = "ecas02"
  "vm_ids" = "i-06e869a7c822b34f6"
  "vm_instance_type" = "t2.medium"
  "vm_private_dns" = "ip-10-10-10-88.eu-central-1.compute.internal"
  "vm_private_ips" = "10.10.10.88"
  "vm_subnet_ids" = "subnet-0102e530cae8471ed"
}

Appreciate any help anyone can give.

Thanks!

2

Hello,

I think the most direct way to process the data structure you want is to first break the individual maps down into a list of the final objects you want. Here we nest 2 iterations, the outer will iterate based on the number of ids to create the number of objects we want, and the inner will add each key and the indexed value to that object.

  obj_list = [ for i, _ in var.test_data_map["vm_ids"] :
    { for k, v in var.test_data_map :
      (k) => v[i]
    }
  ]

Once you have the objects, then you can map them based on the desired attribute value:

  obj_map = { for obj in local.obj_list :
    (obj["vm_ids"]) => obj
  }

This results in the output:

{
  "i-013d3a8efb84bfdbf" = {
    "vm_amis" = "ami-067436817bb0256c1"
    "vm_availability_zones" = "eu-central-1a"
    "vm_hostnames" = "ecas01"
    "vm_ids" = "i-013d3a8efb84bfdbf"
    "vm_instance_type" = "t2.medium"
    "vm_private_dns" = "ip-10-10-10-37.eu-central-1.compute.internal"
    "vm_private_ips" = "10.10.10.37"
    "vm_subnet_ids" = "subnet-075464d83a0ce90f8"
  }
  "i-06e869a7c822b34f6" = {
    "vm_amis" = "ami-067436817bb0256c1"
    "vm_availability_zones" = "eu-central-1b"
    "vm_hostnames" = "ecas02"
    "vm_ids" = "i-06e869a7c822b34f6"
    "vm_instance_type" = "t2.medium"
    "vm_private_dns" = "ip-10-10-10-88.eu-central-1.compute.internal"
    "vm_private_ips" = "10.10.10.88"
    "vm_subnet_ids" = "subnet-0102e530cae8471ed"
  }
}
3

Wow, thank you so much.

Seems so effortless - I would never have got to that point,… hopefully next time with this lesson I’ll have more of a chance :slightly_smiling_face:

Would never have thought to base the number of iterations on the list length. I did attempt to use range/length and pass into v[] but you can’t use number type in a for expression

> [ for i, _ in var.test_data_map["vm_ids"] : i ]
[
  0,
  1,
]

What purpose do the round brackets around (k) and (obj["vm_ids"]) serve? in:

{ for k, v in var.test_data_map : (k) => v[i] }
{ for obj in local.obj_list : (obj["vm_ids"]) => obj }

Again, thank you!

4

Oh yeah, I forgot to remove the extra parentheses, you can test it out and verify there is no difference in output. You do need them to disambiguate between a literal map key and an interpolated value, but here they are just left over as I was reducing the problem.