Nested looping inside EOT

Hello All,

I have a map defined like this in my variables.tf

variable "appdetails" {
  type = map(any)
  default = {
    sample = {
      appid        = "123456",
      appname      = "sampleapp",
      environments = ["env1", "env2", "prod"],
      channel      = "alert_channel"
    }
 }
}

Terraform code to create some DataDog URL Monitors like this in my main.tf

locals {
  monitor_pairs = flatten([
    for monitor in var.appdetails : [
      for env in monitor.environments : {
        appid        = monitor.appid
        appname      = monitor.appname
        env          = env
        channel      = monitor.channel
      }
    ]
  ])
}
...
resource "datadog_synthetics_test" "url_monitoring" {
  for_each   = { for p in local.monitor_pairs : "${p.appid}-${p.env}" => p }
  ...
  message = (
    each.value.env == "prod" ? <<EOT
     The Prod Endpoint for the ${title(each.value.appname)} (${each.value.appid}) is not returning 200.        
        Endpoint: [https://${each.value.appname}.<url>)
        
        {{#is_alert}}
        ${each.value.channel}
        {{/is_alert}}
    EOT
    : <<EOT
    The Non-Prod Endpoint for the ${title(each.value.appname)} (${each.value.appid}) is not returning 200.        
        Endpoint: [https://${each.value.appname}.<url>)
        
        {{#is_alert}}
        ${each.value.channel}
        {{/is_alert}}
        EOT
  )
.....
}

My question is now I want to include a region as well in the Endpoint like this

Endpoint: [https://${each.value.appname}.<Region>.<url>) where Region can be say AWS us-east-1,us-east-2,us-west-2 etc.

Is there a way to do that?

Thanks in advance.

Hi @srinath,

Is the region defined in some other variable that you’ve not shown in the comment here? You can interpolate any strings you need into a string template like this one, but I don’t see anything like a region in the examples you shared so far so I can’t give a specific example.

It might help also if you could share an example of what some full results of this template might look like in different sitautions, so we can work backwards from those intended results to a general rule to produce those and other similar results.

Hi @apparentlymart ,

I was thinking of defining it like this in the variables.tf file

  type = map(any)
  default = {
    sample = {
      appid        = "123456",
      appname      = "sampleapp",
      environments = ["env1", "env2", "prod"],
      channel      = "alert_channel",
      region        = ["us-east-1","us-east-2"]
    }
 }
}

However, I was not sure how that’d work out in the flatten loop now that I have introduced another list. The final result that I expect to see is a separate URL monitor that looks for a 200 HTTP status code for URL’s like this -
Endpoint: [https://${each.value.appname}.<url>) - Separate monitor
Endpoint: [https://${each.value.appname}.us-east-1.<url>) - Separate monitor
Endpoint: [https://${each.value.appname}.us-east-2.<url>) - Separate monitor

Does that make it clearer?
Thanks in advance.

So is your goal to have one instance of datadog_synthetics_test" "url_monitoring" for every combination of appid, env, and region? Or do you instead want to keep your current structure of one instance per appid and env but to include more than one endpoint URL inside your message string?

You mentioned in your title “inside EOT” which I understood to mean "in the template for message" (which you’ve defined using <<EOT syntax), so I’m wondering if your intent is to include repetition in the template or to simply have even more instances of the resource that each only have a single endpoint.

Hi @apparentlymart ,

Sorry for not being clear, this is what I wanted to achieve if that can be done,

variables.tf like this

type = map(any)
  default = {
    sample = {
      appid        = "123456",
      appname      = "sampleapp",
      environments = ["qa", "staging", "prod"],
      channel      = "alert_channel",
      region        = ["us-east-1","us-east-2"]
    }
 }
}

main.tf (existing code works fine without adding region)

locals {
  monitor_pairs = flatten([
    for monitor in var.appdetails : [
      for env in monitor.environments : {
        # for region in region: # This threw a region is tuple with 2 elements error
        appid        = monitor.appid
        appname      = monitor.appname
        env          = env
        channel      = monitor.channel
       # region = monitor.region
      }
    ]
  ])
}

resource "datadog_synthetics_test" "url_monitoring" {
  for_each   = { for p in local.monitor_pairs : "${p.appid}-${p.env}" => p }
  ...
  message = (
    each.value.env == "prod" ? <<EOT
The Prod Endpoint for the ${title(each.value.appname)} (${each.value.appid}) is not returning 200.        
        Endpoint: [https://${each.value.appname}.<url>)
EOT
    : <<EOT
    The Non-Prod Endpoint for the ${title(each.value.appname)} (${each.value.appid}) is not returning 200.        
        Endpoint: [https://${each.value.appname}.<url>)
        
        {{#is_alert}}
        ${each.value.channel}
        {{/is_alert}}
        EOT
  )

Pseudocode below for what I want to see if it can be done:

# qa does not need region
if each.value.env == "qa"
message = (...
Endpoint: [https://${each.value.appname}-qa.<url>) 
...)
# staging and prod need separate monitors for each region
...) or
if each.value.env == "staging" or each.value.env == "prod"
message = (...
Endpoint: [https://${each.value.appname}-<region>-staging.<url>) 
message = (...
Endpoint: [https://${each.value.appname}-<region>-prod.<url>)
...) 
# staging and prod need separate monitors without region in it as well
if each.value.env == "staging" or each.value.env == "prod"
message = (...
Endpoint: [https://${each.value.appname}-staging.<url>) 
...) or
message = (...
Endpoint: [https://${each.value.appname}-prod.<url>)
...) 

Hope that makes sense now. Thanks in advance.

Hi @apparentlymart

Any ideas on how this can be done?

Thanks

Hi @srinath,

It seems like the requirement is to declare one datadog_synthetics_test.url_monitoring instance for each distinct tuple of appid, env, and region, with the special case that a null value for regions is treated as a special “non-region” entry.

The typical first step for a requirement like this is to transform the given data structure into a different shape where there’s one element per instance to declare, which we can do with local values:

variable "app_details" {
  type = map(object({
    app_id       = string
    app_name     = string
    environments = set(string)
    regions      = set(string)
    channel      = string
  }))
}

locals {
  # First we'll normalize the input so that
  # a null "regions" becomes a set with one
  # element, since otherwise having no regions
  # would lead to us generating no tuples at
  # all. An empty string region name signifies
  # a region-agnostic endpoint here.
  app_details = [
    for k, d in var.app_details : {
      key          = k
      app_id       = d.app_id
      app_name     = d.app_name
      environments = d.environments
      regions      = coalesce(d.regions, toset([""]))
      channel      = d.channel
    }
  ]

  app_monitors = flatten([
    for d in local.app_details : [
      for env in d.environments : [
        for reg in d.regions : {
          key         = "${d.app_id}:${env}%{ if reg != "" }:${reg}%{ endif }"
          app_id      = d.app_id
          app_name    = d.app_name
          environment = env
          region      = reg != "" ? reg : null
          channel     = d.channel
          url         = "https://${d.app_name}%{ if reg != "" }-${reg}%{ endif }-${env}.<url>"
          is_prod     =  env == "prod"
        }
      ]
    ]
  ])
}

resource "datadog_synthetics_test" "url_monitoring" {
  for_each = { for m in local.app_monitors : m.key => m }

  request_definition {
    method = "GET"
    url    = each.value.url
  }

  # ...

  message = <<-EOT
    The ${ each.value.is_prod ? "Prod" : "Non-Prod" } Endpoint for the ${title(each.value.app_name)} (${each.value.app_id}) is not returning 200.
    Endpoint: [${each.value.url}]

    {{#is_alert}}
    ${each.value.channel}
    {{/is_alert}}
  EOT
}

This is largely the same as your original approach of flattening the two-level hierarchy of apps and environments. I added in a third level of hierarchy for the regions in a similar way, but also factored out some of the rules for constructing strings that may or may not contain regions just to keep that extra complexity as isolated as possible.

The main “special” thing here compared to the normal flatten pattern is the fact that I treated a null regions as a single region with an empty name, which then allows a region-agnostic application to work even though by normal flatten rules you’d otherwise end up with no elements at all for that application in the case where there are no regions.

Here’s an example of how to set the variable with some of the regions set to null to activate that special behavior:

  app_details = {
    sampleapp_regional = {
      app_id       = "123456"
      app_name     = "sampleapp"
      environments = ["staging", "prod"]
      channel      = "alert_channel"
      regions      = ["us-east-1", "us-east-2"]
    }
    sampleapp_nonregional = {
      app_id       = "123456"
      app_name     = "sampleapp"
      environments = ["qa"]
      channel      = "alert_channel"
      regions      = null
    }
  }

Notice that the “sampleapp” is split into two entries here so that the qa incarnation of it can have regions = null and thus activate the no-regions behavior for it.

This is quite a long example and I’ve not tested it, so I expect I’ve probably made at least one typo in there somewhere. If you get an error when you try this and you’re not sure how to resolve it, let me know and I’ll see if I can improve the example.

Thanks @apparentlymart , I will try it out and get back. Need some time in understanding what you have written and implement the same.

Hi @apparentlymart ,

Trying to test your code to make sure initial testing looks good but running into an the following errors.

╷
│ Error: Unsupported attribute
│ 
│   on main.tf line 20, in locals:
│   20:       app_id       = d.app_id
│ 
│ This object does not have an attribute named "app_id".
╵
╷
│ Error: Unsupported attribute
│ 
│   on main.tf line 21, in locals:
│   21:       app_name     = d.app_name
│ 
│ This object does not have an attribute named "app_name".
╵
╷
│ Error: Unsupported attribute
│ 
│   on main.tf line 22, in locals:
│   22:       environments = d.environments
│ 
│ This object does not have an attribute named "environments".
╵
╷
│ Error: Unsupported attribute
│ 
│   on main.tf line 23, in locals:
│   23:       regions      = coalesce(d.regions, toset([""]))
│ 
│ This object does not have an attribute named "regions".
╵
╷
│ Error: Unsupported attribute
│ 
│   on main.tf line 24, in locals:
│   24:       channel      = d.channel
│ 
│ This object does not have an attribute named "channel".

Any ideas why?

variables.tf

variable "app_details" {
  #   type = map(object({
  #     app_id       = string
  #     app_name     = string
  #     environments = set(string)
  #     regions      = set(string)
  #     channel      = string
  #   }))
  type = map(any)
  default = {
    app_details = {
      sampleapp_regional = {
        app_id       = "123456"
        app_name     = "sampleapp"
        environments = ["staging", "prod"]
        channel      = "alert_channel"
        regions      = ["us-east-1", "us-east-2"]
      }
      sampleapp_nonregional = {
        app_id       = "123456"
        app_name     = "sampleapp"
        environments = ["qa"]
        channel      = "alert_channel"
        regions      = null
      }
    }
  }

}
variable "private_locations" {
  type = list(string)
  default = [
    "default"
  ]
}

main.tf

terraform {
  required_providers {
    datadog = {
      source  = "DataDog/datadog"
      version = "3.10.0"
    }
  }
}
locals {

  # First we'll normalize the input so that
  # a null "regions" becomes a set with one
  # element, since otherwise having no regions
  # would lead to us generating no tuples at
  # all. An empty string region name signifies
  # a region-agnostic endpoint here.
  app_details = [
    for k, d in var.app_details : {
      key          = k
      app_id       = d.app_id
      app_name     = d.app_name
      environments = d.environments
      regions      = coalesce(d.regions, toset([""]))
      channel      = d.channel
    }
  ]

  app_monitors = flatten([
    for d in local.app_details : [
      for env in d.environments : [
        for reg in d.regions : {
          key         = "${d.app_id}:${env}%{if reg != ""}:${reg}%{endif}"
          app_id      = d.app_id
          app_name    = d.app_name
          environment = env
          region      = reg != "" ? reg : null
          channel     = d.channel
          url         = "https://${d.app_name}%{if reg != ""}-${reg}%{endif}-${env}.<url>"
          is_prod     = env == "prod"
        }
      ]
    ]
  ])
}

resource "datadog_synthetics_test" "url_monitoring" {
  for_each = { for m in local.app_monitors : m.key => m }

  request_definition {
    method = "GET"
    url    = each.value.url
  }

  # ...
  name      = "Test"
  type      = "api"
  locations = var.private_locations
  status    = "live"
  message   = <<-EOT
    The ${each.value.is_prod ? "Prod" : "Non-Prod"} Endpoint for the ${title(each.value.app_name)} (${each.value.app_id}) is not returning 200.
    Endpoint: [${each.value.url}]

    {{#is_alert}}
    ${each.value.channel}
    {{/is_alert}}
  EOT
}

Thanks

Hi @srinath,

It looks like you wrote type = map(any) instead of the exact type constraint I suggested, which therefore meant that Terraform was unable to give you feedback on your mistake here, which was that you’ve set default to be a map of map of objects instead of a map of objects.

If you use the exact type constraint that I suggested, Terraform should tell you that the default value you specified isn’t of the correct type, and then it should work if you correct it to have a suitable type, like this:

variable "app_details" {
  type = map(object({
    app_id       = string
    app_name     = string
    environments = set(string)
    regions      = set(string)
    channel      = string
  }))
  default = {
    sampleapp_regional = {
      app_id       = "123456"
      app_name     = "sampleapp"
      environments = ["staging", "prod"]
      channel      = "alert_channel"
      regions      = ["us-east-1", "us-east-2"]
    }
    sampleapp_nonregional = {
      app_id       = "123456"
      app_name     = "sampleapp"
      environments = ["qa"]
      channel      = "alert_channel"
      regions      = null
    }
  }
}

Any time you use the any placeholder in a type constraint you will prevent Terraform from verifying that your input is of the correct type, and so it will tend to return less useful errors like the ones you saw here.

Thanks for pointing that out @apparentlymart - I will try your suggestion.

Thank you very much @apparentlymart , the code seems to be doing what it is supposed to do! Really appreciate your inputs on this.