Pulling values from 2x variables to build another

dal30011 · May 22, 2024, 5:03pm

Hi all,

I’m trying to iterate 2x variables and turn them into something that is suitable for another resource I have to use in a for_each/dynamic block. I think I’m getting lost because of the different types - one is a map containing lists of objects, the other is a map of objects containing various types.

Here’s an example:

targets = {
  functions = [
    {
      name   = "f1"
      group  = "rgp3"
    },
    {
      name   = "f2"
      group  = "rgp3"
    }
  ]

  databases = [
    {
      name             = "sql1"
      group            = "rgp1"
    }
    {
      name            = "sql2"
      group            = "rgp1"
    }
  ]
}

metrics = {
  functions = {
      folder = "Tests"
      panels = [
        {
          alert            = true
          metric           = "executionCount"
        },
        {
          alert            = true
          metric           = "Connections"
        }
      ]
    },

  databases = {
    folder = "Tests"
    panels = [
      {
        alert            = true
        metric           = "CPU%"
      },
      {
        alert            = true
        metric           = "Memory%"
      }
    ]
  }

I’m trying to get to a point where I have an element to iterate over for each individual target in var.targets that also contains info about each of the corresponding metrics it has for var.metrics.

For example, something like this:

new_var = [
  {
    name = "f1"
    resource_group = "rgp3"
    type = "functions"
    metric = "executionCount"
  },
  {
    name = "f1"
    resource_group = "rgp3"
    type = "functions"
    metric = "Connections"
  },
  {
    name = "f2"
    resource_group = "rgp3"
    type = "functions"
    metric = "executionCount"
  },
  {
    name = "f2"
    resource_group = "rgp3"
    type = "functions"
    metric = "Connections"
  },
  {
    name = "sql1"
    resource_group = "rgp1"
    type = "databases"
    metric = "CPU%"
  },
  {
    name = "sql1"
    resource_group = "rgp1"
    type = "databases"
    metric = "Memory%"
  },

  etc...

]

I can get pretty close but I’m reaching my terraform expression knowledge limit

ExtelligenceIT · May 24, 2024, 9:20am

Here you go. Paste the below sample code block into a main.tf in a bare directory and run a terraform plan / apply over it to play around.

I extended your ‘desired output’ example to include the remaining two objects and show that the calculated new_var matches the desired_output by way of using the setsubtract() function to validate they match.

The output type of new_var is:

type(new_var)
new_var = tuple([
  object({
    metric : string,
    name : string,
    resource_group : string,
    type : string,
  }),
  ...
])

Sample Code

locals {
  targets = {
    functions = [
      {
        name  = "f1"
        group = "rgp3"
      },
      {
        name  = "f2"
        group = "rgp3"
      }
    ]

    databases = [
      {
        name  = "sql1"
        group = "rgp1"
      },
      {
        name  = "sql2"
        group = "rgp1"
      }
    ]
  }
  metrics = {
    functions = {
      folder = "Tests"
      panels = [
        {
          alert  = true
          metric = "executionCount"
        },
        {
          alert  = true
          metric = "Connections"
        }
      ]
    },

    databases = {
      folder = "Tests"
      panels = [
        {
          alert  = true
          metric = "CPU%"
        },
        {
          alert  = true
          metric = "Memory%"
        }
      ]
    }
  }

  desired_output = [
    {
      metric         = "executionCount"
      name           = "f1"
      resource_group = "rgp3"
      type           = "functions"

    },
    {
      metric         = "Connections"
      name           = "f1"
      resource_group = "rgp3"
      type           = "functions"

    },
    {
      metric         = "executionCount"
      name           = "f2"
      resource_group = "rgp3"
      type           = "functions"

    },
    {
      metric         = "Connections"
      name           = "f2"
      resource_group = "rgp3"
      type           = "functions"

    },
    {
      metric         = "CPU%"
      name           = "sql1"
      resource_group = "rgp1"
      type           = "databases"

    },
    {
      metric         = "Memory%"
      name           = "sql1"
      resource_group = "rgp1"
      type           = "databases"

    },
    {
      metric         = "CPU%"
      name           = "sql2"
      resource_group = "rgp1"
      type           = "databases"

    },
    {
      metric         = "Memory%"
      name           = "sql2"
      resource_group = "rgp1"
      type           = "databases"

    }
  ]
}

locals {

  new_var = flatten([for targets, targets_values in local.targets : [
    for target in targets_values : [
      for metric in local.metrics[targets].panels : {
        name           = target.name
        resource_group = target.group
        type           = targets
        metric         = metric.metric
      }
    ]
  ]])
}

output "desired_output" {
  value = local.desired_output
  
}

output "new_var" {
  value = local.new_var

}

output "difference" {
  value = setsubtract(local.desired_output, local.new_var)
}

Output for `new_var`

  + new_var        = [
      + {
          + metric         = "CPU%"
          + name           = "sql1"
          + resource_group = "rgp1"
          + type           = "databases"
        },
      + {
          + metric         = "Memory%"
          + name           = "sql1"
          + resource_group = "rgp1"
          + type           = "databases"
        },
      + {
          + metric         = "CPU%"
          + name           = "sql2"
          + resource_group = "rgp1"
          + type           = "databases"
        },
      + {
          + metric         = "Memory%"
          + name           = "sql2"
          + resource_group = "rgp1"
          + type           = "databases"
        },
      + {
          + metric         = "executionCount"
          + name           = "f1"
          + resource_group = "rgp3"
          + type           = "functions"
        },
      + {
          + metric         = "Connections"
          + name           = "f1"
          + resource_group = "rgp3"
          + type           = "functions"
        },
      + {
          + metric         = "executionCount"
          + name           = "f2"
          + resource_group = "rgp3"
          + type           = "functions"
        },
      + {
          + metric         = "Connections"
          + name           = "f2"
          + resource_group = "rgp3"
          + type           = "functions"
        },
    ]

Hope that helps!

Happy Terraforming

dal30011 · May 28, 2024, 8:06am

@ExtelligenceIT - that’s great, thanks, really appreciate your reply. Was actually a bit easier than I was making it in the end… Had the typical moment of “should have seen that” when I saw your solution .

Are there any good resources on getting a thorough understanding of Terraform expressions/for expressions in particular? I’ve had a look at the official docs but I find it’s a bit light on detail/explained example, beyond some really simple ones.

ExtelligenceIT · May 29, 2024, 11:48am

Thanks for the feedback @dal30011

I find the approach of stripping out the expression into a local, in a ‘bare’ module as I have shown in this example makes things a lot more straightforward to then play with.

A tip is to not add functions such as flatten/merge/etc. until you have got all of the attributes extracted from your source that you need, as it makes it easier to track where things are happening.

Other than that I try and approach it in an incremental manner. e.g.:

I need to process each target in the map
For each of the targets I need to process the list of objects
What values do I need from the current target values
What values do I need from the targets key
For each object in the list I need to access values from metrics that can be indexed using the current index value of targets
What values do I need from the metrics
Get rid of all of the unneeded nesting

  flatten([for targets, targets_values in local.targets : [    <--1,7
    for target in targets_values : [                           <--2
      for metric in local.metrics[targets].panels : {          <--5
        name           = target.name                           <--3
        resource_group = target.group                          <--3
        type           = targets                               <--4
        metric         = metric.metric                         <--6
      }
    ]
  ]])

There will have been some extra tweaks along the way to adjust the datatypes coming out at each stage. e.g… As the desired output was a list of objects the bracket at the point of steps 2-4 at for target in targets_values : would have been { not the [ it turned into when adding step 5.

Sometimes it will take a couple of attempts, or a night’s sleep but overall the above approach tends to work for me.

HTH!

system · July 30, 2024, 11:49am

This topic was automatically closed 62 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
How to iterate over map of objects? Terraform	1	5118	July 8, 2022
[TF 0.12.x] Syntax for resource creation by non-numeric index For-loop - Can this be done in current version of Terraform? Terraform	3	1040	June 26, 2019
For_each over a set of objects, creating keys from an object property that is a list of strings Terraform	2	2155	June 17, 2023
Reusable values inside of for_each Terraform	2	1055	October 15, 2020
For_each and conditions Terraform	0	261	August 4, 2023

Pulling values from 2x variables to build another

Sample Code

Output for new_var

Related topics

Output for `new_var`