Terraform data massage

Terraform
1

Just pulling my hair …

How to merge one field’ values into a list, while keep other fields untouched? Say, I have a list like this:

[
{
“k8s” = “a”
“ns” = “ns1”
“sa” = “a”
},
{
“k8s” = “a”
“ns” = “ns2”
“sa” = “a”
},
]

And I’d like to generate a final list like:
[
{
“k8s” = “a”
“ns” = [“ns1”, “ns2”]
“sa” = “a”
},
]

Straight for loop doesn’t work, Any recommendations? Thanks,

Best,
David

2

It’s not possible to accomplish this transform in any reasonably nice way using Terraform - it’s beyond the capabilities of the language, and it’s better to do this in some kind of generation step that outputs files before Terraform is run.

Technically there is a not-nice way to do it:

locals {
  foo = your input here

  foo_merged = [
    for key_it in toset([
      for element_it in local.foo : {
        k8s = element_it.k8s
        sa  = element_it.sa
      }
    ])
    :
    {
      k8s = key_it.k8s
      sa  = key_it.sa
      ns  = [
        for match_it in local.foo :
        match_it.ns
        if match_it.k8s == key_it.k8s && match_it.sa == key_it.sa
      ]
    }
  ]
}

It works, but at what cost? It’s seriously unreadable, extends poorly to additional fields, and requires a separate loop over ALL the input data for each unique key, so scales poorly too.

3

Actually, just after posting this, I came across a fairly obscure feature I’d never used before, which allows some of the issues to be addressed:

  foo_merged_2 = [
    for grouping_it in {
      for element_it in local.foo :
      "${element_it.k8s} ${element_it.sa}" => element_it...
    }
    :
    {
      k8s = grouping_it[0].k8s
      sa  = grouping_it[0].sa
      ns  = [for item_it in grouping_it : item_it.ns]
    }
  ]

This is a fair bit better - it resolves the poor scaling characteristics I was complaining about above.

It’s still fairly hard to read, and comes with the new downside of needing to be able to merge all of your grouping fields into a single string (I’m assuming I can rely on k8s and sa never containing space characters in their own values here.)

4

That’s perfect!, solve the problem so elegantly,

The area that I don’t understand is the grouping values together with ‘…’ operator, documents said that it can only be used inside a function, like min([1,2,3,-1]...), what’s its use after element_it?

Thanks a lot,

5

Got it, it’s called grouping feature, Thanks a lot.

6

Indeed, the one set of punctuation ... is actually two completely different language constructs, depending on where it appears.