Using data returned by jsondecode and iterate over the results in a for_each loop

Terraform
1

Here is a sample of my json data that adequately mocks my use case.

{      

   "environmentName":"",

   "environmentDetails1":"",

   "environmentDetails2":"",

   "environmentDetails3":"",

   "servers":[

      {         

         "type":"windows",

         "vmName":"server1",

         "cpus":1,

         "ram":11111,

         "diskThinProvisioned":true,

         "disk":{            

            "diskSize":11111,

            "diskLabel":"diskLabel"

         },

         "networks":[

            {                              

               "ipAddress":"1.1.1.1",

               "netmask":"111.111.111"

            }

         ]                  

      },

      {

         "type":"linux",

         "vmName":"server2",

         "cpus":1,

         "ram":11111,

         "diskThinProvisioned":true,         

         "disk":{            

            "diskSize":11111,

            "diskLabel":"diskLabel"            

         },

         "networks":[

            {                              

               "ipAddress":"1.1.1.1",

               "netmask":"111.111.111"               

            }

         ]         

      }      

   ] 

}

I intend on using the http data source once I have this working but for now, I’m using the
local_file data source to fetch my json file.

data "local_file" "getJsonFile" {

    filename = "${path.module}/configData.json"

}

I have a local variable defined that will perform the decoding of my json data.

locals {    

    configData = jsondecode(data.local_file.getJsonFile.content)

}

Note: The data type returned from the the jsondecode results is an object.

Next I would like to iterate over the servers array in a for_each loop. In my json data example, I have two servers but in reality, that array may hold dozens.

Naturally, If I wanted to iterate over any collection in a for_each loop, I would simply select my traversal object (my collection). However, If I access the servers like configData.servers the data type is now a tuple. From my reading and testing, you must use a map or a set of strings in a for_each loop. I’ve spent a absurd amount of time using Terraforms type conversion functions in conjunction with changing the entire format of my json data to make this work. Nothing has worked for me so far which Is why I’m reaching out here for some assistance. Any help would be greatly appreciated!

A major bonus for any insight on how to add a filter either in the local variable configData or in the for_each loop itself where I can distinguish the different server types. For example: if server.type == "linux" or if server.type == "windows"

2

You can convert the data to a set right in the for_each statement:
for_each = toset(local.configData.servers)

This example prints a list of full server objects if the server type is linux or windows; I’m not sure if this is enough to get you started but I hope it helps:

output "servers" {
  value = [
    for s in toset(local.configData.servers): s if s.type == "linux" || s.type = "windows"
  ]
}
3

@mildwonkey Thanks for the reply. I had arrived at a similar solution previously but ended up scrapping it due to the for_each limitations with sets. When doing a to toset() on local.configData.servers I end up with a list of objects. The for_each argument is expecting either a map or a list of strings. Due to this limitation on the for_each argument, It seems I can only use a map in my particular situation.

4

hm! does tolist() work, then?

5

Converting my local.configData.servers to a list works as expected however, I’m not able to use that list in a for_each loop. Here is a reference. The only allowed datatypes in a for_each loop are maps and set of strings.

6

Right, I’m sorry, you said that already.

You could use for in a local to get the list of servers, and then use the value of that to index into var.configData.servers[each.key] in your resource?

7

That’s an idea. Could you possibly show me a simple example?

8

One option is to convert the data structure to a map (in locals), and you can use that for your for_each expression:

servermap = {for s in tolist(local.config.servers): s.vmName => s if s.type == "linux" || s.type == "windows"}

9

Thanks @mildwonkey. I appreciate your working with me on this.

I’m glad you suggested this because I tried this before except I didn’t use a tolist() in the for_each loop and I ended up facing a new problem. The problem is that in my resource where I’m using the for_each loop, each element is now named with a key of whatever s.vmName is. For example, using the sample json in my OP, I have two servers and in order to access the attributes of that first server in my for_each loop, I would need to hard code the vmName value directly into my resource. An example of how I would access the object in my resource (ex: each.value.server1.ram). As you might expect, hardcoding the vmName defeats the purpose of what I’m doing.

While this is technically a different problem from what I posted and possibly deservers it’s own thread, if solved, it does fix my current problem. With that being said, do you have any insight on this new problem?

10

I guess it would depend on how you would rather access objects in your resource! If you’d prefer it to be a list instead of a map (so you use index numbers instead of strings), I think something like this would work (I haven’t tested this):

serverlist = [for s in tolist(local.config.servers): s.vmName if s.type == "linux" || s.type == "windows"]

Then use the serverlist as your for_each, and use that vmName to index into the servermap (at this point there’s likely a more elegant solution that I’m not thinking of, so play around with it!)

11

Back from the holidays and I was able to resolve my issue and wanted to follow up on my post for any future readers.

TLDR: I ended up dropping the for_each loop all together and replacing it with count and indexing into my local variable that had the jsondecoded data.

We were quite disappointed with the limitations of the for_each meta argument. The most frustrating part is that it could not iterate over all Terraform collection types. The key problem I was running into was that JSON objects will never convert to a map while using the jsondecode function regardless of how you format your JSON object. You could of course always convert an object to a map using the tomap() function but I would always run into an issue once I applied the filter using the for expression. The for expression could only output either a tuple or an object which are both unusable in a for_each loop. Again, I could convert to a map using the tomap() function but I would just end up drilling into a tuple once I hit an array in my main data object. At this point I was stuck in an endless “loop” of converting my data to something usable in the for_each loop at which point I threw in the towel and decided to rethink my approach to the problem. I ended up using the count meta argument for my loop instead of the for_each. In my example below, you can see how I’m indexing into my variable from within the module.

locals {
  windowsConfigs = [ for x in jsondecode(data.local_file.getJsonFile.content).configurations : x if x.type == "windows" ]
  linuxConfigs   = [ for x in jsondecode(data.local_file.getJsonFile.content).configurations : x if x.type == "linux" ]
}

    // Example of how it is implemented
    module "myWindowsModule" {

       count = length(local.windowsConfigs)
       hostName = local.windowsConfigs[count.index].vmName
       ...
    }

Thanks @mildwonkey for sticking with me threw this issue xD