Merging complex objects?

Is it possible to do merging on complex objects in the same way you do with maps?

In locals I have this, something similar to what we use for merging maps:

locals {
  merged_additional_tags = merge(
    {
    "primary" = {
      "aas" = {
        "primary" = {}
        "ha" = {}
      }
      "ci" = {
        "primary" = {}
        "ha" = {}
      }
      "db" = {
        "primary" = {}
        "ha" = {}
      }
      "worker" = {
        "primary" = {}
        "ha" = {}
      }
    }
    "dr" = {
      "db" = {}
      "aas" = {}
      "ci" = {}
      "worker" = {}
    }
  },
  var.additional_tags)
}

Then in tfvars to define the values in the additional_tags variable

additional_tags = {
    "primary" = {
      "aas" = {
        "primary" = {
          "testing" = "fake_tag"
         }
      }
   }
}

The end result seems to be the only values that end up in local.merged_additional_tags is what I specified in the tfvars, there is no merging happening. Is this possible with objects? Or is there a better way to do this?

Hi @nickmhankins,

The merge function does only a shallow merge, so the fact that primary is present in var.additional_tags means its value completely replaces the primary object in the base structure.

Implementing a deep merge may be possible using some complex nested for expressions, but in this situation I’d be inclined to try to find a different solution where the merging would be deferred to some later point where we already know we’re working with a leaf map in this structure and can thus focus only on merging the leaf levels. I can’t show an example of that without some additional context to show what your end goal was here, but if you can show what you ultimately intended to do with this merged data structure I may be able to suggest a different approach.

Hi @apparentlymart, is there any chance a deep merge function would be implemented in Terraform anytime soon?

Hi @apparentlymart ,

My intent with the object is to be able to add arbitrary tags at the tfvars level that can be selectively applied to a multitude of modules and resources that live a few layers deep inside other modules (they will also be used to override other tags if needed). I didn’t want to have to specify the entire additional_tags object in the tfvars, only what is needed (but it looks like I may need to in this case). The tags can’t be applied to all resources in the sub-modules.

For example, at the top level, I was thinking I would pass in the tags like this:

tags = merge(local.tags, {additional_tags = local.additional_tags["primary"]["db"]})

local.tags contains maps base_tags and vm_tags.

The additional tags that are passed in are only applied to specific resources in the sub-modules, for example, one of the sub-modules might have something like this:

primary_vm_tags =  merge(var.tags["base_tags"], var.tags["vm_tags"], {"Role" = "HANADB"}, var.tags["additional_tags"]["primary"])

ha_vm_tags = merge(var.tags["base_tags"], var.tags["vm_tags"], {"Role" = "HANADB"}, var.tags["additional_tags"]["ha"])

Here’s what I came up with for a deep merge focused on this specific use-case:


variable "additional_tags" {
  type = map(map(map(map(string))))
  default = {}
}

locals {
  default_tags = {
    primary = {
      aas = {
        primary = {}
        ha      = {}
      }
      ci = {
        primary = {}
        ha      = {}
      }
      db = {
        primary = {}
        ha      = {}
      }
      worker = {
        primary = {}
        ha      = {}
      }
    }
    dr = {
      aas = {
        primary = {}
        ha      = {}
      }
      ci = {
        primary = {}
        ha      = {}
      }
      db = {
        primary = {}
        ha      = {}
      }
      worker = {
        primary = {}
        ha      = {}
      }
    }
  }

  merged_tags = {
    for env, apps in local.default_tags : env => {
      for app, tiers in apps : app => {
        for tier, tags in tiers : tier => merge(
          tags,
          lookup(
            lookup(
              lookup(
                var.additional_tags,
                env, {(app) = {}},
              ),
              app, {(tier) = {}},
            ),
            tier, {},
          )
        )
      }
    }
  }
}

output "merged_tags" {
  value = local.merged_tags
}

This depends on a change coming in 0.12.7 (due soon) to allow lookup to work with nested data structures, so unfortunately it won’t work with the 0.12.6 release that is current at the time I write this, but I did verify it against a build from the latest master branch.

I wrote a .tfvars file containing the following:

additional_tags = {
  primary = {
    db = {
      primary = {
        name = "baz"
      }
    }
  }
}

After applying it, I got the following result:

Outputs:

merged_tags = {
  "dr" = {
    "aas" = {
      "ha" = {}
      "primary" = {}
    }
    "ci" = {
      "ha" = {}
      "primary" = {}
    }
    "db" = {
      "ha" = {}
      "primary" = {}
    }
    "worker" = {
      "ha" = {}
      "primary" = {}
    }
  }
  "primary" = {
    "aas" = {
      "ha" = {}
      "primary" = {}
    }
    "ci" = {
      "ha" = {}
      "primary" = {}
    }
    "db" = {
      "ha" = {}
      "primary" = {
        "name" = "baz"
      }
    }
    "worker" = {
      "ha" = {}
      "primary" = {}
    }
  }
}

Generally when decomposing configuration into modules I try to keep the tag handling centralized in the root module and only pass individual modules the specific leaf set of tags they need, rather than passing around a big data structure of tags for all systems, but this somewhat-awkward nested for expression allows merging the big data structure on the assumption that the structure is always consistently-shaped (it’s always a map(map(map(map(string)))), not a mixture of different types at different levels) and that the default_tags structure defines exhaustively all of the possible environment, application, and tier names that additional_tags is permitted to override.


Regarding the possibility of having a “deep merge” function built in to Terraform: in both Terraform and in other tools we’ve seen functionality like that which seems at first to be pretty simple but then gets overcome by edge-cases. The pertinent Terraform examples are the automatic merging of map variables that was removed in Terraform 0.12 (because it caused more confusion than benefit in practice) and the override files mechanism which is still supported but has grown quite a long and complex list of rules about how different constructs merge together.

I expect we would not try to implement a naive “deep merge” function in Terraform in the near future because it’s likely to fall into the same complexity hole: different folks would have different needs and expectations for what to do in cases where the two data structures don’t have exactly the same shape, whether non-map collections ought to merge or override each other, etc.

We’ll see over time if some common patterns emerge that we could build a function around, but we’d want to scope it very carefully so that it’s easy to predict how it will behave without reading multiple paragraphs of documentation about various caveats and edges. Since the ability to pass around deeply-nested maps in Terraform is still relatively new, we haven’t seen enough real-world examples yet to define what exactly that might look like. In the mean time, nested for expressions are a (admittedly rather awkward) way to implement certain types of nested merging, when the two structures have a fixed shape.

1 Like

I’ve implemented a generallized deepmerge module that you might find useful @nickmhankins and @sleterrier. It’s worked for everything I’ve tested it with, but I could use some more testers! https://github.com/Imperative-Systems-Inc/terraform-modules/tree/master/deepmerge

2 Likes

Since this topic bumped back to the top again :smiley: I guess I should also mention my apparentlymart/javascript provider which you can (if you are using Terraform 0.13 or later) use to do some complicated data structure wrangling using JavaScript code, possibly using Underscore.js:

terraform {
  required_providers {
    javascript = {
      source = "apparentlymart/javascript"

      # Until this provider has a stable release, always select an exact
      # version to avoid adopting new releases that may have breaking changes.
      version = "0.0.1"
    }
  }
}

data "javascript" "merge_tags" {
  source = file("${path.module}/merge_tags.js")
  vars = {
    additionalTags = var.additional_tags
  }
}

Bringing in an entirely new language is a pretty extreme strategy, so I’d consider this to be a last resort rather than something to be employed regularly, but if you do have a particularly hairy data structure wrangling problem then factoring it out into a separate script like this means you can more easily decompose the problem into small custom functions, and use recursion to handle nesting.

1 Like

Seems like there may not be a way to do this [today] with native terraform in just a few lines? The deepmerge module does what we want, but seems like a roundabout approach for merging complex objects

Thanks for sharing this, it’s very useful to be able to modify Kubernetes manifests and other configs that are sourced from somewhere one cannot easily modify. Of course it’s possible to use kustomize in some cases, but it can be much easier to just use your JavaScript provider and avoid shelling out!

@apparentlymart is there a place where we can tally the latest thoughts on deep merge? There we would document various use cases and categorize them as main or edge.

Because right now, it seems to me that 99% of users would be satisfied with merging maps/objects and overwriting lists, and the 1% that aren’t could use a provider like your javascript one. My experience is that merging lists is rarely needed or practical, because for lists you have too many options: insert before, after, try to combine items of the lists based on some unique key, etc. I don’t think anyone can reasonably expect a deepmerge function to support all those options. I don’t know of a language that does, probably because it is too problem-specific and would be impossible to remember (“overwrites lists” is easy to remember; anything else is not).

So I’d really like to see those use cases that you mention in other threads documented somewhere and we should have a discussion about whether they are worth blocking on.

The behavior I mention would not require pages of documentation, but a short note that list are overwritten. Nothing more to say about it.

And one day when HCL supports custom functions (:grin:), such deepmerge could optionally accept a custom function, which would get called with the two nodes being merged and would return a new node, and the user could then write their own merge algorithm for lists (or anything for that matter).

Thanks so much for this deepmerge module. You’ve just saved me hours of banging my head against a wall trying to merge x-amazon-apigateway-integration fragments into an OpenAPI schema. :smiley: