Using local variables in setproduct with for expression

Terraform
1

Hi,

I’m struggling with getting a local variable be parsed as a map (rather than an object) in a setproduct that contains a for expression, where the local variable defined below

locals {
  external_location_service_principal_access = [
    for pair in setproduct(
      local.storage_accounts_flat,
      [
        for service_principal_key, service_principal_object in local.service_principals :
        merge({ service_principal_key = service_principal_key }, service_principal_object)
      ]
    ) :
    merge(pair[0], pair[1])
    if pair[0].databricks_external_location == true
    && length(pair[1].databricks.privileges_external_location) > 0
  ]
}

leads to the following error:

│ Error: Invalid function argument
│ 
│   on imports.tf line 17, in locals:
│   15:     for pair in setproduct(
│   16:       local.storage_accounts_flat,
│   17:       [
│   18:         for service_principal_key, service_principal_object in local.service_principals :
│   19:         merge({ service_principal_key = service_principal_key }, service_principal_object)
│   20:       ]
│   21:     ) :
│     ├────────────────
│     │ while calling setproduct(sets...)
│     │ local.service_principals is object with 3 attributes
│ 
│ Invalid value for "sets" parameter: all elements must be of the same type.

Some clarifications are in order to understand why this is unexpected:

  • The referenced local variable storage_accounts_flat is a list of objects (all of the same type).
  • The local variable service_principals is defined as a map of objects, although it seems Terraform interprets it as an object instead.
  • The second argument in the setproduct intends to build a list of objects out of a map (taking a key-value pair and merging the key into the value)
  • The set product should therefore build the cartesian product of the two lists of objects, merging the object pairs.

I’m already using essentially the same logic to define some resource’s iterated instances in a module, and it works with no problem. Notice that the only difference between the local variable below and the one above is that the one below refers to a module variable service_principals that is explicitly defined as a map of objects via type constraints.

locals {
  external_location_service_principal_access = [
    for pair in setproduct(
      var.storage_accounts_flat,
      [
        for service_principal_key, service_principal_object in var.service_principals :
        merge({ service_principal_key = service_principal_key }, service_principal_object)
      ]
    ) :
    merge(pair[0], pair[1])
    if pair[0].databricks_external_location == true
    && length(pair[1].databricks.privileges_external_location) > 0
  ]
}

resource "databricks_grant" "external_location_service_principal" {
  for_each = {
    for v in local.external_location_service_principal_access :
    "${v.storage_account}.${v.container_name}.${v.service_principal_key}" => v
  }

  external_location = databricks_external_location.this["${each.value.storage_account}.${each.value.container_name}"].id
  principal         = data.databricks_service_principal.this[each.value.service_principal_key].application_id
  privileges        = each.value.databricks.privileges_external_location
}

I need the definition at the top (the one using the local variable service_principals), to iterate an import block targeting the databricks_grant resource. Somehow Terraform is attempting to expand the setproduct arguments too deep until it stumbles with the local.service_principals variable, which does have a different type but is enclosed within a for expression of list type, which should actually be the operand for the setproduct.

2

Hi @camilo-s,

Can you create an example with sample data which shows the problem you are having? Without actual data we can’t tell what Terraform is inferring the types to be, only that the local.storage_accounts_flat value and the result of the for expression don’t result in the same type.

3

Hi @jbardin,

thanks for your quick reply. I’ll try to cook an MWE up this week and post it here, as the actual context is rather convoluted for me to extract the data flowing in.

4

One way you might be able to reduce the problem is to add some explicit type conversion function calls to make sure that Terraform can interpret the types the way you are expecting it to.

For example, based on your error message I would add a tomap call for local.service_principals to assert that this object ought to be able to be interpreted as a map, since you’ve stated that assumption in your list. Although it’s not mentioned in the error message, you might as well also confirm your assertion that local.storage_accounts_flat is either a list or is convertible to one:

    for pair in setproduct(
      tolist(local.storage_accounts_flat),
      [
        for service_principal_key, service_principal_object in tomap(local.service_principals) :
        merge({ service_principal_key = service_principal_key }, service_principal_object)
      ]
    ) :

My hope with making these changes is that you would see a more specific error message that would narrow down the problem some more.

For example, if tomap(local.service_principals) generates an error then the problem reduces only to figuring out why local.service_principals isn’t map-compatible, and so you could ignore the rest of this large expression and focus only on solving that smaller problem.

5

Thanks for the suggestion @apparentlymart.

I did try using tomap to enforce the type conversion. However, Terraform wasn’t able to infer the target types:

│ Error: Invalid function argument
│ 
│   on imports.tf line 18, in locals:
│   18:         for service_principal_key, service_principal_object in tomap(local.service_principals) :
│     ├────────────────
│     │ while calling tomap(v)
│     │ local.service_principals is object with 3 attributes
│ 
│ Invalid value for "v" parameter: cannot convert object to map of any single
│ type.
╵
##[error]Bash exited with code '1'.

The objects in the local.service_principals have some “optional attributes”, but there is no way to declare this inside a local variable, so my guess is that Terraform perceives the objects in the map as incompatible and is consequently unable to do a type conversion. My current workaround is to declare a service principals variable with all necessary type constraints and then enrich it in the service_principals local variable with some computed attributes I need (like IDs).

This works fine, however it’s unintuitive to me why the set product has to dive into the type of the iterator local.service_principals in the second operand since the output type of the for expression is a list anyway.

6

Both lists and sets are collection types and so have the requirement that all of the elements be of the same type. (And that is also true for maps, as we see in this error message.)

The experiment I proposed seems to have confirmed my guess that the service principals object is the root problem here: the attributes of that object are not of consistent types and so the elements of the list you are trying to derive from it would also have inconsistent types, and therefore it fails. Adding tomap just narrowed down where the inconsistency was coming from.

You mentioned “optional attributes”, but that’s an idea that belongs to type constraints (like in the type argument of an input variable declaration) rather than to the concrete types of actual values. Since local values are not subjected to any automatic type conversion at the point of definition, there is no opportunity for optional attributes to be handled there and so you would need to write the local value expression in a way that gets the same effect as a type conversion would have achieved:

  • Wrap the local value definition itself in tomap, so you can be sure that it’s always a valid map whenever used anywhere else. This is equivalent to having an input variable declared as type = map(any), and so is a good start to making sure there is a consistent type across all of the elements.

  • For any attribute that you consider as “optional”, assign a typed null to it to explicitly represent that it is part of the type but omitted for this particular value of the type.

    For example, if you would’ve written optional(string) in a type constraint for this attribute, you would write tostring(null) to get the same placeholder that type conversion would have produced.

If you do both of these things then you should have a valid map which you can then assume has elements of consistent types when you write your for expression as in you original question.

7

Thanks for the very clarifying response @apparentlymart.

I managed to cook an MWE up FWIW:

# mwe.tf

variable "storage_accounts" {
  type = map(object({
    name       = string
    containers = optional(map(string), {})
  }))
  default = {}
}


locals {
  service_principals = {
    sp1 = {
      display_name   = "Service Principal 1"
      storage_access = true
      tags           = { foo = "bar" }
    }
    sp2 = {
      display_name   = "Service Principal 2"
      storage_access = false
    }
    sp3 = {
      display_name   = "Service Principal 3"
      storage_access = true
      tags           = { bar = "biz" }
    }
  }
  storage_accounts_flat = flatten([
    for storage_account_key, storage_account_value in var.storage_accounts :
    [
      for container_key, container_value in storage_account_value.containers :
      {
        storage_account_key  = storage_account_key
        storage_account_name = storage_account_value.name
        container_key        = container_key
      }
    ]
  ])
  storage_account_service_principal_access = [
    for pair in setproduct(
      local.storage_accounts_flat,
      [
        for service_principal_key, service_principal_value in local.service_principals :
        merge(
          { service_principal_key = service_principal_key },
          service_principal_value
        )
      ]
    ) : merge(pair[0], pair[1])
    if pair[1].storage_access
  ]
}

resource "terraform_data" "this" {
  for_each = {
    for v in local.storage_account_service_principal_access :
    "${v.storage_account_key}.${v.container_key}.${v.service_principal_key}" => v.storage_account_name
  }

  input = "${each.key}-${each.value}"
}

# dev.tfvars

storage_accounts = {
  act1 = {
    containers = {
      cont1 = ""
      cont2 = ""
    }
    name = "act1"
  }
  act2 = {
    containers = {
      cont1 = ""
    }
    name = "act2"
  }
  act3 = {
    name = "act3"
  }
}

Proceeding as you suggested fixes the issue. I didn’t try that in my original setting though because I’d have to basically rewrite the whole service_principals local variable, which I wanted to avoid.

8

The error message suggests that the setproduct function expects all elements in the sets provided to be of the same type. In your code, it seems that local.storage_accounts_flat is a set, but the second set (created from local.service_principals) is an object with attributes.

To resolve this issue, you need to make sure that both sets passed to setproduct are of the same type. If you intend to use the keys or values from local.service_principals, you may need to convert them into a set of the same type as local.storage_accounts_flat.

Here’s an example of how you might adjust the code:

locals {
service_principal_objects = [
for service_principal_key, service_principal_object in local.service_principals :
merge({ service_principal_key = service_principal_key }, service_principal_object)
]

external_location_service_principal_access = [
for pair in setproduct(
local.storage_accounts_flat,
toset(local.service_principal_objects)
) :
merge(pair[0], pair[1])
if pair[0].databricks_external_location == true
&& length(pair[1].databricks.privileges_external_location) > 0
]
}