How to do a setproduct from a list and a list of lists?

Terraform
1

Hi there,
I have set of inputs like this:

---
vpc_list:
  main:
    cidr: 10.2.220.0/20
    endpoints_if: [
      ec2,
      logs
    ]
  extra:
    cidr: 10.3.220.0/27
    endpoints_if: [
    ]

# Default Interface Endpoints
vpc_endpoints_if:
  - ec2
  - monitoring
  - kms

and from there, either I can produce two sets of lists:

> keys(var.vpc_list)
[
  "main",
  "extra",
]

and

> [for key in keys(var.vpc_list) : setunion(var.vpc_endpoints_if, var.vpc_list[key]["endpoints_if"])]
[
  [
    "ec2",
    "kms",
    "logs",
    "monitoring",
  ],
  [
    "ec2",
    "kms",
    "monitoring",
  ],
]

or a list of keys and the corresponding list of values:

> zipmap(keys(var.vpc_list), [for key in keys(var.vpc_list) : setunion(var.vpc_endpoints_if, var.vpc_list[key]["endpoints_if"])])
{
  "main" = [
    "ec2",
    "kms",
    "logs",
    "monitoring",
  ]
  "extra" = [
    "ec2",
    "kms",
    "monitoring",
  ]
}

From there, how can I create two individual setproducts, for main and extra each and then combine those two and get some thing like this?

[
  [
    "main",
    "ec2",
  ],
  [
    "main",
    "log",
  ],
  .....
  [
    "extra",
    "ec2",
  ],
  .....
]

so that I can feed that in when create the VPC endpoints:

resource "aws_vpc_endpoint" "if_eps" {
  for_each = {
    for pair in setproduct(--the-magic-recipe--) :
    "${pair[0]}:${pair[1]}" => {
      vpc_key = pair[0]
      ep_name = pair[1]
    }
  }
  ....

Am I thinking right or there is a better way of doing it? Any help would be really appreciated.

-S

2

Hi @dsantanu,

In this case you seem to have a potentially different set of second-level keys for each top-level key, so this seems like it might be more of a job for flatten than for setproduct. Something like this, perhaps:

flatten([
  for vpc_key, obj in var.vpc_list : [
    for endpoint_key in obj.endpoints_if : {
      vpc_key     = vpc_key
      endpoint_key = endpoint_key
    }
  ]
])
3

thanks @apparentlymart for the very prompt answer. I think we are getting somewhere better but if I run the above on my config, I just get this:

[
  {
    "endpoint_key" = "ec2"
    "vpc_key" = "main"
  },
  {
    "endpoint_key" = "logs"
    "vpc_key" = "main"
  },
]

It’s doing some for main VPC and not for extra at all.

I think, I understand why, as the obj.endpoints_if is empty for extra but actually I need to merge var.vpc_endpoints_if and obj.endpoints_if to get the resultant list of the Endpoints for any given VPC.

-S

4

Okay, I think I have figured out: I had to do this instead:

flatten([
  for vpc_key, obj in var.vpc_list : [
    for endpoint_key in setunion(var.vpc_endpoints_if, obj.endpoints_if) : {
      vpc_key     = vpc_key
      endpoint_key = endpoint_key
    }
  ]
])

Now I’ll see how can I feed that as for_each to the aws_vpc_endpoint resource.
Many thanks for giving m the clue :+1:

-San

5

Great! I’m glad you figured out the part I missed, and sorry I didn’t catch that detail on my first read.

A typical final part of this "flatten for for_each" pattern is to put a for expression in the for_each to reshape it into being a map of objects rather than a list of objects, like this:

  for_each = {
    for obj in local.flattened :
    "${obj.vpc_key}:${obj.endpoint_key}" => obj
  }

The string template in there is how you can tell Terraform what shape you want to use for these compound keys, because the endpoint keys alone are not unique across all of the endpoints you need to declare.

I used a colon separator here just because it’s a typical convention I follow, but you can actually use whatever string representation you like as long as it will be unique for all of the pairs.

6

thanks again, @apparentlymart!
I actually did a very similar thing in the end:

for_each = {
    for pair in local.if_eps :
    "${pair.vpc_key}:${pair.ep_name}" => {
      vpc_key = pair.vpc_key
      ep_name = pair.ep_name
    }
  }

mainly b’cuz I was already using things like each.value.vpc_key and each.value.ep_name in my previous version of the code, where endpoints list was coming straight only from var.vpc_endpoints_if. So just changed the for_eachpart keeping rest of the code as it is. You are as helpful as always; couldn’t have done that quickly without your help. :bowing_man: