Merging complex objects?

Is it possible to do merging on complex objects in the same way you do with maps?

In locals I have this, something similar to what we use for merging maps:

locals {
  merged_additional_tags = merge(
    {
    "primary" = {
      "aas" = {
        "primary" = {}
        "ha" = {}
      }
      "ci" = {
        "primary" = {}
        "ha" = {}
      }
      "db" = {
        "primary" = {}
        "ha" = {}
      }
      "worker" = {
        "primary" = {}
        "ha" = {}
      }
    }
    "dr" = {
      "db" = {}
      "aas" = {}
      "ci" = {}
      "worker" = {}
    }
  },
  var.additional_tags)
}

Then in tfvars to define the values in the additional_tags variable

additional_tags = {
    "primary" = {
      "aas" = {
        "primary" = {
          "testing" = "fake_tag"
         }
      }
   }
}

The end result seems to be the only values that end up in local.merged_additional_tags is what I specified in the tfvars, there is no merging happening. Is this possible with objects? Or is there a better way to do this?

Hi @nickmhankins,

The merge function does only a shallow merge, so the fact that primary is present in var.additional_tags means its value completely replaces the primary object in the base structure.

Implementing a deep merge may be possible using some complex nested for expressions, but in this situation I’d be inclined to try to find a different solution where the merging would be deferred to some later point where we already know we’re working with a leaf map in this structure and can thus focus only on merging the leaf levels. I can’t show an example of that without some additional context to show what your end goal was here, but if you can show what you ultimately intended to do with this merged data structure I may be able to suggest a different approach.

Hi @apparentlymart, is there any chance a deep merge function would be implemented in Terraform anytime soon?

Hi @apparentlymart ,

My intent with the object is to be able to add arbitrary tags at the tfvars level that can be selectively applied to a multitude of modules and resources that live a few layers deep inside other modules (they will also be used to override other tags if needed). I didn’t want to have to specify the entire additional_tags object in the tfvars, only what is needed (but it looks like I may need to in this case). The tags can’t be applied to all resources in the sub-modules.

For example, at the top level, I was thinking I would pass in the tags like this:

tags = merge(local.tags, {additional_tags = local.additional_tags["primary"]["db"]})

local.tags contains maps base_tags and vm_tags.

The additional tags that are passed in are only applied to specific resources in the sub-modules, for example, one of the sub-modules might have something like this:

primary_vm_tags =  merge(var.tags["base_tags"], var.tags["vm_tags"], {"Role" = "HANADB"}, var.tags["additional_tags"]["primary"])

ha_vm_tags = merge(var.tags["base_tags"], var.tags["vm_tags"], {"Role" = "HANADB"}, var.tags["additional_tags"]["ha"])

Here’s what I came up with for a deep merge focused on this specific use-case:


variable "additional_tags" {
  type = map(map(map(map(string))))
  default = {}
}

locals {
  default_tags = {
    primary = {
      aas = {
        primary = {}
        ha      = {}
      }
      ci = {
        primary = {}
        ha      = {}
      }
      db = {
        primary = {}
        ha      = {}
      }
      worker = {
        primary = {}
        ha      = {}
      }
    }
    dr = {
      aas = {
        primary = {}
        ha      = {}
      }
      ci = {
        primary = {}
        ha      = {}
      }
      db = {
        primary = {}
        ha      = {}
      }
      worker = {
        primary = {}
        ha      = {}
      }
    }
  }

  merged_tags = {
    for env, apps in local.default_tags : env => {
      for app, tiers in apps : app => {
        for tier, tags in tiers : tier => merge(
          tags,
          lookup(
            lookup(
              lookup(
                var.additional_tags,
                env, {(app) = {}},
              ),
              app, {(tier) = {}},
            ),
            tier, {},
          )
        )
      }
    }
  }
}

output "merged_tags" {
  value = local.merged_tags
}

This depends on a change coming in 0.12.7 (due soon) to allow lookup to work with nested data structures, so unfortunately it won’t work with the 0.12.6 release that is current at the time I write this, but I did verify it against a build from the latest master branch.

I wrote a .tfvars file containing the following:

additional_tags = {
  primary = {
    db = {
      primary = {
        name = "baz"
      }
    }
  }
}

After applying it, I got the following result:

Outputs:

merged_tags = {
  "dr" = {
    "aas" = {
      "ha" = {}
      "primary" = {}
    }
    "ci" = {
      "ha" = {}
      "primary" = {}
    }
    "db" = {
      "ha" = {}
      "primary" = {}
    }
    "worker" = {
      "ha" = {}
      "primary" = {}
    }
  }
  "primary" = {
    "aas" = {
      "ha" = {}
      "primary" = {}
    }
    "ci" = {
      "ha" = {}
      "primary" = {}
    }
    "db" = {
      "ha" = {}
      "primary" = {
        "name" = "baz"
      }
    }
    "worker" = {
      "ha" = {}
      "primary" = {}
    }
  }
}

Generally when decomposing configuration into modules I try to keep the tag handling centralized in the root module and only pass individual modules the specific leaf set of tags they need, rather than passing around a big data structure of tags for all systems, but this somewhat-awkward nested for expression allows merging the big data structure on the assumption that the structure is always consistently-shaped (it’s always a map(map(map(map(string)))), not a mixture of different types at different levels) and that the default_tags structure defines exhaustively all of the possible environment, application, and tier names that additional_tags is permitted to override.


Regarding the possibility of having a “deep merge” function built in to Terraform: in both Terraform and in other tools we’ve seen functionality like that which seems at first to be pretty simple but then gets overcome by edge-cases. The pertinent Terraform examples are the automatic merging of map variables that was removed in Terraform 0.12 (because it caused more confusion than benefit in practice) and the override files mechanism which is still supported but has grown quite a long and complex list of rules about how different constructs merge together.

I expect we would not try to implement a naive “deep merge” function in Terraform in the near future because it’s likely to fall into the same complexity hole: different folks would have different needs and expectations for what to do in cases where the two data structures don’t have exactly the same shape, whether non-map collections ought to merge or override each other, etc.

We’ll see over time if some common patterns emerge that we could build a function around, but we’d want to scope it very carefully so that it’s easy to predict how it will behave without reading multiple paragraphs of documentation about various caveats and edges. Since the ability to pass around deeply-nested maps in Terraform is still relatively new, we haven’t seen enough real-world examples yet to define what exactly that might look like. In the mean time, nested for expressions are a (admittedly rather awkward) way to implement certain types of nested merging, when the two structures have a fixed shape.