Untangle a data structure

Terraform
1

Hi all, any help would be appreciated.

First thing I’d like to know is with HCL and expressions is it possible to massage data from one data structure to another? Well, obviously it is because I’ve done it here but It always seems quite cumbersome compared to a fully featured language, some times including multiple wrangling steps. Maybe this is because the intentions of the expressions or terraform itsself is being stretched. It seems like the target state of my desired data structure is always just out of reach.

I have an ugly but necessary data structure like this

[
   {
     "a": {
        "prop1": 1,
        "prop2": 2,
        "prop3": 3
      }
     "b": {
        "prop1": 1,
        "prop2": 2,
        "prop3": 3
      }
   },
   {
    "c": {
        "prop1": 1,
        "prop2": 2,
        "prop3": 3
    }
    "d": {
        "prop1": 1,
        "prop2": 2,
        "prop3": 3
    }
   },
]

an unfortunate consequence of dynamically building the strucutre and using it with several modules. I need to pass these data to a module using a for_each for each set of property blob. The the actual names for a b c and d are arbitrary and really the properties are what’s important.

What I need is this:

   {
     "a": {
        "prop1": 1,
        "prop2": 2,
        "prop3": 3
      }
     "b": {
        "prop1": 1,
        "prop2": 2,
        "prop3": 3
      }
    "c": {
        "prop1": 1,
        "prop2": 2,
        "prop3": 3
    }
    "d": {
        "prop1": 1,
        "prop2": 2,
        "prop3": 3
    }
}

I’ve tried quite a few ways of doing it each scarier than the last :smiley:. In terms of taking work back to my team or documenting loops and data structures I struggle with how the language wants me to use it vs what I want it to do.

toset(flatten([for assignment in local.managed_identity_policy_assignment_roles: [ for properties in assignment: properties ] ] ) )

This got me

{
 {
        "prop1": 1,
        "prop2": 2,
        "prop3": 3
  },
  {
        "prop1": 1,
        "prop2": 2,
        "prop3": 3
    }
... etc 
}

But a module doesn’t take a set of objects, it needs the map. Is there something glaringly obvious I’m missing here apart from my own desire to use hcl like python!

2

Fundamentally, HCL is an expression language only. It’s a lot more limited than a full programming language, and whilst it’s good for glueing data from one thing into another thing in simpler ways, more complex transformations exceed what it can handle, and tend to become painfully unreadable before they get to that point.

However, in the specific case you outline, it can be accomplished really simply:

merge(your_input_structure...)

NOTE: This will silently discard entries if you have duplicate keys.

3

If you do need an error in case of duplicate keys, it’s possible, but a fair bit more messy:

  { for item in flatten(
    [for inner in your_input_structure :
      [for k, v in inner : { k = k, v = v }
  ]]) : item.k => item.v }
4

Wild man, as elegant of a solution as I could’ve hoped. There’s misleadingly a lot more going on here than meets the eye. Due to the nature of how the original strucutre if built, there are no duplicate keys, so this should work perfectly.

And yes, hard to read, misleading and lacking context are usually how handling data ends up in terraform. Do you have any advice on getting more to grips with the internals of the expression language, it looks very rego like when it comes to for.

5

I don’t think I have any particularly special advice - all I know is just from the docs and experience, built up over time.

But, if I had to pick a few particular language constructs to single out as particularly worth learning in detail:

  • The merge and flatten functions
  • For expressions
    • Take care to notice that input can be list-like or map-like, and the output can also be independently list-like or map-like. So, there are 4 varieties: list>list, list>map, map>list, map>map.
    • Notice that they also support if clauses for filtering
    • Notice that they also support a syntax involving ... for grouping
  • Notice that function calls also have a special syntax using ... to mean something different, which was instrumental in the solution
  • Note that nesting multiple for expressions inside a call to flatten, is a standard idiom whenever you need for_each to iterate over multiple dimensions - to the point that this even has a special mention with examples in the flatten function documentation.
1 Like