Pulling values from 2x variables to build another

Hi all,

I’m trying to iterate 2x variables and turn them into something that is suitable for another resource I have to use in a for_each/dynamic block. I think I’m getting lost because of the different types - one is a map containing lists of objects, the other is a map of objects containing various types.

Here’s an example:

targets = {
  functions = [
    {
      name   = "f1"
      group  = "rgp3"
    },
    {
      name   = "f2"
      group  = "rgp3"
    }
  ]

  databases = [
    {
      name             = "sql1"
      group            = "rgp1"
    }
    {
      name            = "sql2"
      group            = "rgp1"
    }
  ]
}


metrics = {
  functions = {
      folder = "Tests"
      panels = [
        {
          alert            = true
          metric           = "executionCount"
        },
        {
          alert            = true
          metric           = "Connections"
        }
      ]
    },

  databases = {
    folder = "Tests"
    panels = [
      {
        alert            = true
        metric           = "CPU%"
      },
      {
        alert            = true
        metric           = "Memory%"
      }
    ]
  }

I’m trying to get to a point where I have an element to iterate over for each individual target in var.targets that also contains info about each of the corresponding metrics it has for var.metrics.

For example, something like this:

new_var = [
  {
    name = "f1"
    resource_group = "rgp3"
    type = "functions"
    metric = "executionCount"
  },
  {
    name = "f1"
    resource_group = "rgp3"
    type = "functions"
    metric = "Connections"
  },
  {
    name = "f2"
    resource_group = "rgp3"
    type = "functions"
    metric = "executionCount"
  },
  {
    name = "f2"
    resource_group = "rgp3"
    type = "functions"
    metric = "Connections"
  },
  {
    name = "sql1"
    resource_group = "rgp1"
    type = "databases"
    metric = "CPU%"
  },
  {
    name = "sql1"
    resource_group = "rgp1"
    type = "databases"
    metric = "Memory%"
  },

  etc...

]

I can get pretty close but I’m reaching my terraform expression knowledge limit :frowning:

Here you go. Paste the below sample code block into a main.tf in a bare directory and run a terraform plan / apply over it to play around.

I extended your ‘desired output’ example to include the remaining two objects and show that the calculated new_var matches the desired_output by way of using the setsubtract() function to validate they match.

The output type of new_var is:

type(new_var)
new_var = tuple([
  object({
    metric : string,
    name : string,
    resource_group : string,
    type : string,
  }),
  ...
])

Sample Code

locals {
  targets = {
    functions = [
      {
        name  = "f1"
        group = "rgp3"
      },
      {
        name  = "f2"
        group = "rgp3"
      }
    ]

    databases = [
      {
        name  = "sql1"
        group = "rgp1"
      },
      {
        name  = "sql2"
        group = "rgp1"
      }
    ]
  }
  metrics = {
    functions = {
      folder = "Tests"
      panels = [
        {
          alert  = true
          metric = "executionCount"
        },
        {
          alert  = true
          metric = "Connections"
        }
      ]
    },

    databases = {
      folder = "Tests"
      panels = [
        {
          alert  = true
          metric = "CPU%"
        },
        {
          alert  = true
          metric = "Memory%"
        }
      ]
    }
  }

  desired_output = [
    {
      metric         = "executionCount"
      name           = "f1"
      resource_group = "rgp3"
      type           = "functions"

    },
    {
      metric         = "Connections"
      name           = "f1"
      resource_group = "rgp3"
      type           = "functions"

    },
    {
      metric         = "executionCount"
      name           = "f2"
      resource_group = "rgp3"
      type           = "functions"

    },
    {
      metric         = "Connections"
      name           = "f2"
      resource_group = "rgp3"
      type           = "functions"

    },
    {
      metric         = "CPU%"
      name           = "sql1"
      resource_group = "rgp1"
      type           = "databases"

    },
    {
      metric         = "Memory%"
      name           = "sql1"
      resource_group = "rgp1"
      type           = "databases"

    },
    {
      metric         = "CPU%"
      name           = "sql2"
      resource_group = "rgp1"
      type           = "databases"

    },
    {
      metric         = "Memory%"
      name           = "sql2"
      resource_group = "rgp1"
      type           = "databases"

    }
  ]
}

locals {

  new_var = flatten([for targets, targets_values in local.targets : [
    for target in targets_values : [
      for metric in local.metrics[targets].panels : {
        name           = target.name
        resource_group = target.group
        type           = targets
        metric         = metric.metric
      }
    ]
  ]])
}

output "desired_output" {
  value = local.desired_output
  
}

output "new_var" {
  value = local.new_var

}

output "difference" {
  value = setsubtract(local.desired_output, local.new_var)
}

Output for new_var

  + new_var        = [
      + {
          + metric         = "CPU%"
          + name           = "sql1"
          + resource_group = "rgp1"
          + type           = "databases"
        },
      + {
          + metric         = "Memory%"
          + name           = "sql1"
          + resource_group = "rgp1"
          + type           = "databases"
        },
      + {
          + metric         = "CPU%"
          + name           = "sql2"
          + resource_group = "rgp1"
          + type           = "databases"
        },
      + {
          + metric         = "Memory%"
          + name           = "sql2"
          + resource_group = "rgp1"
          + type           = "databases"
        },
      + {
          + metric         = "executionCount"
          + name           = "f1"
          + resource_group = "rgp3"
          + type           = "functions"
        },
      + {
          + metric         = "Connections"
          + name           = "f1"
          + resource_group = "rgp3"
          + type           = "functions"
        },
      + {
          + metric         = "executionCount"
          + name           = "f2"
          + resource_group = "rgp3"
          + type           = "functions"
        },
      + {
          + metric         = "Connections"
          + name           = "f2"
          + resource_group = "rgp3"
          + type           = "functions"
        },
    ]

Hope that helps!

Happy Terraforming

@ExtelligenceIT - that’s great, thanks, really appreciate your reply. Was actually a bit easier than I was making it in the end… Had the typical moment of “should have seen that” when I saw your solution :smiley:.

Are there any good resources on getting a thorough understanding of Terraform expressions/for expressions in particular? I’ve had a look at the official docs but I find it’s a bit light on detail/explained example, beyond some really simple ones.

1 Like

Thanks for the feedback @dal30011

I find the approach of stripping out the expression into a local, in a ‘bare’ module as I have shown in this example makes things a lot more straightforward to then play with.

A tip is to not add functions such as flatten/merge/etc. until you have got all of the attributes extracted from your source that you need, as it makes it easier to track where things are happening.

Other than that I try and approach it in an incremental manner. e.g.:

  1. I need to process each target in the map
  2. For each of the targets I need to process the list of objects
  3. What values do I need from the current target values
  4. What values do I need from the targets key
  5. For each object in the list I need to access values from metrics that can be indexed using the current index value of targets
  6. What values do I need from the metrics
  7. Get rid of all of the unneeded nesting
  flatten([for targets, targets_values in local.targets : [    <--1,7
    for target in targets_values : [                           <--2
      for metric in local.metrics[targets].panels : {          <--5
        name           = target.name                           <--3
        resource_group = target.group                          <--3
        type           = targets                               <--4
        metric         = metric.metric                         <--6
      }
    ]
  ]])

There will have been some extra tweaks along the way to adjust the datatypes coming out at each stage. e.g… As the desired output was a list of objects the bracket at the point of steps 2-4 at for target in targets_values : would have been { not the [ it turned into when adding step 5.

Sometimes it will take a couple of attempts, or a night’s sleep :slight_smile: but overall the above approach tends to work for me.

HTH!