TERRAFORM (0.12) - Nested For-In or other technique to augment an object in a map

I have a map that looks like this:

stuff = {
  object-1 = {
    id                    = "1"
    main_value            = "blue"
    additional_in_values  = ["red"]
    additional_out_values = ["green"]
    more_values           = "foo"
  },
  object-2 = {
    id                    = "2"
    main_value            = "green"
    additional_in_values  = ["blue"]
    additional_out_values = ["red"]
    more_values           = "fee"
  },
  object-3 = {
    id                    = "3"
    main_value            = "red"
    additional_in_values  = ["green"]
    additional_out_values = ["blue"]
    more_values           = "fie"
  }
}

The essence of the challenge is to be able to freely iterate over an object in a map and selectively add, remove and modify attributes of the object without having to explicitly enumerate them.

The specific use case is to augment each object by either inserting the “main_value” into “additional_in_values” and “additional_out_values”, or much better, combining the main_value with “additional_in_values” and “additional_out_values” to form two new pairs called “inbound_all” and “outbound_all”. There are additional use cases that involve inserting new key-value pairs in the object or dropping named key-value pairs (for example additional_out could be dropped if we had “outbound_all”).

A successful result would look like:

transformed-stuff = {
  object-1 = {
    id                    = "1"
    main_value            = "blue"
    additional_in_values  = ["red"]
    additional_out_values = ["green"]
    inbound_all           = ["blue", "red"]   # concat main_value and additional_in_values
    outbound_all          = ["blue", "green"] # concat main_value and additional_out_values
    more_values           = "foo"
  },
  object-2 = {
    id                    = "2"
    main_value            = "green"
    additional_in_values  = ["blue"]
    additional_out_values = ["red"]
    inbound_all           = ["green", "blue"]
    outbound_all          = ["green", "red"]
    more_values           = "fee"
  },
  object-3 = {
    id                    = "3"
    main_value            = "red"
    additional_in_values  = ["green"]
    additional_out_values = ["blue"]
    inbound_all           = ["red", "green"]
    outbound_all          = ["red", "blue"]
    more_values           = "fie"
  }
}

The key constraint is to do these insert or concatenate operations in a way that preserves the existing object attributes without explicitly naming each attribute. Naming each attribute would work fine but would be unmaintainable as each of the objects actually has many more attributes than shown, and the attributes come and go, so a solution that explicitly enumerates won’t do, per the “unmaintainable” transform in the sample code below.

It seems like it should be possible in the loop to implement logic like:

(pseudocode)
maintainable = for stuff_key, stuff_details in var.stuff :
 stuff_key => {
    stuff_detail # just insert the existing detail item in the new detail
    inbound_all     = concat([stuff_details.main_value], stuff_details.additional_in_values) # insert a new element
 }

Or even a solution that selectively drops an element of a given value

Any ideas? I’d think a nested for…in loop should work but my own experiments are failing. If it were possible the inner for…in would iterate over stuff_details and selectively act on each existing item, as well as inserting the new items after

Code is:

variable "stuff" {
  type = map(object({
    id                    = string
    main_value            = string
    additional_in_values  = list(string)
    additional_out_values = list(string)
    more_values           = string
  }))
}

locals {
  unmaintainable = {
    for stuff_key, stuff_details in var.stuff :
      stuff_key => {
        id = stuff_details.id
        main_value = stuff_details.main_value
        additional_out_values = stuff_details.additional_out_values
        additional_in_values = stuff_details.additional_in_values
        more_values = stuff_details.more_values
        inbound_all     = concat([stuff_details.main_value], stuff_details.additional_in_values)
        outbound_all    = concat([stuff_details.main_value], stuff_details.additional_out_values)
      }
   }

  transformed-stuff = {
    for stuff_key, stuff_details in var.stuff :
    stuff_key => {
      inbound_all     = concat([stuff_details.main_value], stuff_details.additional_in_values)
      outbound_all    = concat([stuff_details.main_value], stuff_details.additional_out_values)
    }
  }
}


output "stuff" {
  value = var.stuff
}

output "unmaintainable" {
    value = local.unmaintainable
}

output "transformed-stuff" {
  value = local.transformed-stuff
}

The original value is at the top of the post and can be placed in terraform.tfvars

The not-satisfactory output of the code in its current state is:

Outputs:

stuff = {
  "object-1" = {
    "additional_in_values" = [
      "red",
    ]
    "additional_out_values" = [
      "green",
    ]
    "id" = "1"
    "main_value" = "blue"
    "more_values" = "foo"
  }
  "object-2" = {
    "additional_in_values" = [
      "blue",
    ]
    "additional_out_values" = [
      "red",
    ]
    "id" = "2"
    "main_value" = "green"
    "more_values" = "foo"
  }
  "object-3" = {
    "additional_in_values" = [
      "green",
    ]
    "additional_out_values" = [
      "blue",
    ]
    "id" = "3"
    "main_value" = "red"
    "more_values" = "foo"
  }
}
transformed-stuff = {
  "object-1" = {
    "inbound_all" = [
      "blue",
      "red",
    ]
    "outbound_all" = [
      "blue",
      "green",
    ]
  }
  "object-2" = {
    "inbound_all" = [
      "green",
      "blue",
    ]
    "outbound_all" = [
      "green",
      "red",
    ]
  }
  "object-3" = {
    "inbound_all" = [
      "red",
      "green",
    ]
    "outbound_all" = [
      "red",
      "blue",
    ]
  }
}
unmaintainable = {
  "object-1" = {
    "additional_in_values" = [
      "red",
    ]
    "additional_out_values" = [
      "green",
    ]
    "id" = "1"
    "inbound_all" = [
      "blue",
      "red",
    ]
    "main_value" = "blue"
    "more_values" = "foo"
    "outbound_all" = [
      "blue",
      "green",
    ]
  }
  "object-2" = {
    "additional_in_values" = [
      "blue",
    ]
    "additional_out_values" = [
      "red",
    ]
    "id" = "2"
    "inbound_all" = [
      "green",
      "blue",
    ]
    "main_value" = "green"
    "more_values" = "foo"
    "outbound_all" = [
      "green",
      "red",
    ]
  }
  "object-3" = {
    "additional_in_values" = [
      "green",
    ]
    "additional_out_values" = [
      "blue",
    ]
    "id" = "3"
    "inbound_all" = [
      "red",
      "green",
    ]
    "main_value" = "red"
    "more_values" = "fee"
    "outbound_all" = [
      "red",
      "blue",
    ]
  }
}

You can get somewhat close to your goal using:

output "transformed_stuff" {
  value = { for k, stuff_details in var.stuff :
    k => merge(
      stuff_details,
      {
        inbound_all  = concat([stuff_details.main_value], stuff_details.additional_in_values)
        outbound_all = concat([stuff_details.main_value], stuff_details.additional_out_values)
      }
    )
  }
}

I can’t think of a nice way to remove an attribute without it getting horribly messy, though. At that point, you’re probably better off just deciding to set the attributes to null, and cope with that in the code using them.

Well, I suppose if I really had to implement the removals - say, removing the additional_* bits after they’d been combined - it could be done like:

output "transformed_stuff" {
  value = { for k, augmented_stuff_details in
    { for k, stuff_details in var.stuff :
      k => merge(
        stuff_details,
        {
          inbound_all  = concat([stuff_details.main_value], stuff_details.additional_in_values)
          outbound_all = concat([stuff_details.main_value], stuff_details.additional_out_values)
        }
      )
  } : k => { for k2, v2 in augmented_stuff_details : k2 => v2 if !startswith(k2, "additional_") } }
}

but this is getting really hard to understand now.

I think it’s a pretty good solution overall, I’ll hack on it a bit and see what comes together. There’s not a lot of clean clear examples for comprehensions in tf, and this is incredibly helpful.

startswith() isn’t part of 0.12,but I think substr(offset, len) should do – and in fact, does do fine in your example.

I think the pruning of attributes would be less confusing if done in a second transform, but for some reason the compound expression:

value = { for k, augmented_stuff_details in
    { for k, stuff_details in var.stuff :
      k => merge(
        stuff_details,
        {
          inbound_all  = concat([stuff_details.main_value], stuff_details.additional_in_values)
          outbound_all = concat([stuff_details.main_value], stuff_details.additional_out_values)
        }
      )
  } : k => { for k2, v2 in augmented_stuff_details : k2 => v2 if substr(k2,0,3) != "add" } }

works just fine, but the easier to read twostep:


  transformed-stuff = { for k, stuff_details in var.stuff :
    k => merge(
      stuff_details,
      { 
        inbound_all  = concat([stuff_details.main_value], stuff_details.additional_in_values)
        outbound_all = concat([stuff_details.main_value], stuff_details.additional_out_values)
      }
    )
  }

  pruned-stuff = {
    for transformed_key, transformed_value in local.transformed-stuff :
      transformed_key => transformed_value if substr(transformed_key, 0, 11) != "additional_"
  }

fails to prune and simply reiterates the source map:

pruned-stuff = {
  "object-1" = {
    "additional_in_values" = [
      "red",
    ]
    "additional_out_values" = [
      "green",
    ]
    "id" = "1"
    "inbound_all" = [
      "blue",
      "red",
    ]
    "main_value" = "blue"
    "more_values" = "foo"
    "outbound_all" = [
      "blue",
      "green",
    ]
  }

etc.

Am I missing something obvious?

We’re well past the point of “good enough” and thanks for it! But if there’s an easy way to make the solution and example easily comprehensible (if you’ll forgive the pun) it gets closer to an example of a general solution of being able to shape data structures as easily in TF as in procedural languages.

0.12 is really old now, and upgrading is only going to get harder as knowledge about legacy Terraform 0.x concepts starts to fade from the Internet over time.

OOI, is there anything specific blocking you moving up to 1.x?

I don’t think there is a general solution, as Terraform lacks any way to define chunks of re-usable logic.

As a case-study, one of my more horrible bits of Terraform code at work has a ~10 line complicated expression for converting duration strings like “10m”, “3y”, etc. to time in seconds, which has to be copy-pasted for every single value I need to apply it to.

It’s also impossible to do recursive processing of nested data structures, for example.

I find that one of the important skills for using Terraform, is to know when it’s time to stop using Terraform because your problem has outgrown it.

At that point, you’re iterating over and testing the value of the keys "object-1", etc. to see whether they start with additional_. You’ve stripped off the extra layer of iterating through the properties of the values.

Time, legacy and testing in a big organization. But we’re at least planning it :slight_smile:

Ah. And I’m guessing your point about It’s also impossible to do recursive processing of nested data structures, for example. means I can’t do what I’ve tried to do all along, which is:

  1. Take a map of values where the values are key value pairs
  2. Iterate over the key value pairs and reshape, prune, add to them
  3. Make a new map with the new values, probably with zipmap

Is that really not possible? I understand there’s eventually a time to quit, but it seems so simple, obvious and out of reach

I don’t really understand what you mean here. Perhaps an example would help?

Isn’t that basically what we’ve been discussing earlier in this topic?

I cannot think of any scenario in which the zipmap function is useful in Terraform. It requires you to build the keys and values as separate lists, which feels much more awkward than a comprehension.

One thing you absolutely can’t do in Terraform is recursively apply an algorithm. For example someone recently wanted to flatten a deeply nested structure of maps into one map with keys represented as dot-separated paths. There’s just no way to do that transform in Terraform expression language.

For situations where you really can’t avoid recursive algorithms, I wrote a provider that acts as an escape hatch to JavaScript: apparentlymart/javascript.

It being published in my own namespace rather than in the hashicorp namespace reflects that this is my own personal project rather than a HashiCorp project. I consider it generally “done” now and am not intending to work on it further, but it should already have enough functionality to pass in a data structure, do some computation on it, and return a new data structure.

Because it’s an unofficial provider it’ll be pretty annoying to use on Terraform v0.12 (you’d need to install it manually) but if you manage to upgrade to Terraform v0.13 or later then you’ll be able to auto-install it along with any other providers your configuration already depends on.

I made this only as a last resort for situations where it’s not practical to do something simpler. I’d always suggest trying to simplify the problem first, such as by having your module expect input that is already in the most appropriate data structure for how it will use the data.

Yes, pretty much what we’ve been talking about. Here’s a sample `map of values where the values are key value pairs, from the original problem statement:

stuff = {
  object-1 = {
    id                    = "1"
    main_value            = "blue"
    additional_in_values  = ["red"]
    additional_out_values = ["green"]
    more_values           = "foo"
  },
  object-2 = {
    id                    = "2"
    main_value            = "green"
    additional_in_values  = ["blue"]
    additional_out_values = ["red"]
    more_values           = "fee"
  },
  object-3 = {
    id                    = "3"
    main_value            = "red"
    additional_in_values  = ["green"]
    additional_out_values = ["blue"]
    more_values           = "fie"
  }
}

So, given that structure, we can get

stuff_keys = keys(stuff)
stuff_values = values(stuff)

Now, our goal is to add, subtract and mutate the key, value pairs in stuff_values. For example, one instance of stuff_values from the example above would be:

{
    id                    = "1"
    main_value            = "blue"
    additional_in_values  = ["red"]
    additional_out_values = ["green"]
    more_values           = "foo"
  },

and what I was looking for was to take that stuff_values structure and add, delete or mutate items. So for example the value above might become:

{
    id                    = "1"
    main_value            = "blue"
    inbound_all           = ["blue", "red"]   # concat main_value and additional_in_values
    outbound_all          = ["blue", "green"] # concat main_value and additional_out_values
    more_values           = "foo"
  }

where we’ve removed the additional_*, created the outbound_all and inbound all. Without explicit enumeration of each key/value pair, just changes to the key/value pairs we want to affect. Let’s call the result stuff_values_transformed

That done we create our transformed structure by using a zipmap function with the original keys and the stuff_values_transformed value.

Clear enough? It seems easy and intuitive, but I’ve had no luck creating stuff_values_transformed out of stuff_values without resorting to explicit enumeration.

Comparisons to sliced bread come immediately to mind. Thanks so much for this! It does seem to me that the overall problem is soluble with a zipmap (see recent response to @maxb) but if it isn’t, the javascript provider will do just fine.

Upgrading to 0.13 should not be a problem. I’m going to try to take the install all the way to 1.3, and we’ll see how that goes, but don’t know of anything between 0.13 and 1.3 that will make more than a cosmetic difference.

Appreciate the provider!