Merge multiple json files from fileset

Terraform
1

I am attempting to merge multiple json files.
each file contains the same structure but with different values:
file1:

{
      appservice = [
          {
              + name     = "appserviceplan1"
              + os_type  = "Linux"
              + sku_name = "B1"
              apps     = [
                  {
                     name       = "app1"
                   }
                ]
            },
        ]
 },

file2

{
      appservice = [
          {
              name     = "appserviceplan1"
              os_type  = "Linux"
              sku_name = "B1"
              apps     = [
                  {
                     name       = "app2"
                   }
                ]
            },
        ]
 },

the code for fetching the files and my attempt at merging them:

  json_files = fileset("../../", "**/*infrastructure.json")
  terraforminfrastructure  = [for f in local.json_files : jsondecode(file("../../${f}"))]

this results containing 2 objects each with appservice.
I want to be able to merge the 2 json files so that app service would contain both web apps, but recognize that the appservice name is identical.

my desired result would look like this:

{
      appservice = [
          {
              name     = "appserviceplan1"
              os_type  = "Linux"
              sku_name = "B1"
              apps     = [
                  {
                     name       = "app1"
                   },
                  {
                     name       = "app2"
                   }
                ]
            },
        ]
 },

if anyone can help me with a solution a would be very grateful

2

This merging looks pretty complicated to me, as you have potential issues where there could be mismatches of os_type and sku_name when merging items with the same name.

Because of the complexity of the merge, I would suggest this is beyond what is reasonable to handle within the Terraform expression language, and is better handled in a procedural language capable of definition of functions, storing intermediate results, and producing custom error messages at various steps in the process. This would need to be run separately to write a consolidated JSON file for Terraform to read, before running Terraform.

Note: I’m not saying that this merge would be impossible to do in Terraform expressions… just that it would be some very hard to understand code, that might have some odd behaviour in edge cases, and would likely result in less-than-helpful error messages when supplied with bad input.

3

I broadly agree with @maxb here that the shape of these JSON objects is not well-suited to the transformation you want to make to them, and so any Terraform-based solution to this will include a lot more transformation logic than I’d typically want in a Terraform configuration.

If you do want to deal with this in Terraform then I would suggest breaking the problem into at least two separate steps, where the first step is to transform the individual documents into a data structure that is better-suited to the merging operation you want to perform.

A start on that would be to translate these from just a list of objects into a map of lists of objects where they are grouped by the app service name, perhaps like this:

locals {
  all_appservices = flatten(local.terraforminfrastructure[*].appservice)

  appservices_by_name = tomap({
    for as in local.all_appservices : as.name => as...
  })
}

This would produce a local.appservices_by_name data structure shaped something like this:

tomap({
  appserviceplan1 = [
    {
      name     = "appserviceplan1"
      os_type  = "Linux"
      sku_name = "B1"
      apps = [
        { name = "app1" },
      ]
    },
    {
      name     = "appserviceplan1"
      os_type  = "Linux"
      sku_name = "B1"
      apps = [
        { name = "app2" },
      ]
    },
  ]
})

This data structure at least has all of the entries for a particular name grouped together, so that the remaining problem is to reduce each of those groups of objects into a single object using whatever rules you want to use to deal with conflicts like the ones @maxb mentioned.

With that said, if it’s possible to either manually or automatically reshape these inputs into a form closer to how Terraform will consume them before you run Terraform then I think that would typically be preferable. Although Terraform can do data transforms like this, it’s not Terraform’s strength and it’s instead focused more on relatively straightforward projections where the source data is mostly congruent with the resources you want to describe.

4

Thank you both for your inputs, i agree that it may not be terraforms strength and perhaps handling this outside of terraform is the better choice. The purpose of this post was to see if I was unaware of a quick method of joining them. a special thanks to apparentlymart for suggesting a method of approach.