Experiment Feedback: Optional attribute keys should not be included in variable value unless specified

Terraform
1

I am a huge fan of and avid user of the module_variable_optional_attrs optional parameter for custom objects. After using it for a while, I believe it would be better to omit optional object keys if theyre not passed explicitly instead of including the object with a null value.

Here is a use case. I have a variable type object:

variable "cells_definition" {
  description = "Nested map where the key is a region you want to enable and keys referring to resource arns to enable. Services enabled: `elasticloadbalancing`, `autoscaling`, `lambda`. Example below:"
  type = map(object({
    elasticloadbalancing = optional(string)
    autoscaling          = optional(string)
    lambda               = optional(string)
  }))
}

The value might look like this:

  cells_definition = {
    us-west-2 = {
      elasticloadbalancing = "arn:aws:elasticloadbalancing:us-west-2:<>:loadbalancer/app/<>"
      autoscaling          = "arn:aws:autoscaling:us-west-2:<>:autoScalingGroup:*:autoScalingGroupName/<>
    }
    us-east-1 = {
      elasticloadbalancing = "arn:aws:elasticloadbalancing:us-east-1:<>:loadbalancer/app/<>"
      autoscaling          = "arn:aws:autoscaling:us-east-1:<>:autoScalingGroup:*:autoScalingGroupName/<>"
    }
  }

In my module I attempt to construct a list of all the services passed (the keys) with:

service_list = setintersection(flatten([for k, v in var.cells_definition : keys(v)]))

and the output looks like:

setintersection(flatten([for k, v in var.cells_definition : keys(v)]))
toset([
  "autoscaling",
  "elasticloadbalancing",
  "lambda",
])

Lambda is included even though its not passed in cell_definitions because its included as a nulled value in each iterable:

setintersection(flatten([for k, v in var.cells_definition : v]))
toset([
  {
    "autoscaling" = <>
    "elasticloadbalancing" = <>
    "lambda" = tostring(null)
  },

Anyways, just my 2 cents. It would be better to omit the value

2

FYI i have a workaround atm to use nested for loops

service_list = setintersection(flatten([for _, cell_definition in var.cells_definition : [for service_name, arn in cell_definition : arn != null ? service_name : null]]))

3

Hi @drewmullen1,

If I’m following your worked example correctly, I believe that this is behaving as intended today, though I understand you’re suggesting that you’d prefer it to behave differently!

I think the crux of the matter here is that a value can only be said to conform to the type constraint you specified if it has all of the attributes, because otherwise it would be of a different type with a different set of attributes. Terraform has a structural type system, which means that types are defined by their shape rather than by their name, and so the purpose of the optional attributes experiment is to allow you to create an attribute which conforms to the type without having to explicitly null out all of the attributes. This is essentially the same as how Terraform treats optional attributes in a resource block: omitting them is exactly the same as setting them to null, and if you were to use keys with a resource object you’d see it behave in the same way, returning all of the attribute names regardless of whether they are set to null.

I do understand that for your particular situation you’d rather those attributes not exist at all. I don’t really see a path to Terraform supporting that as a first-class feature because it would drastically change how Terraform thinks about types, and thus it would have significant consequences beyond just the use-case you wanted to meet here.

A different way to get the result you wanted here would be to derive a new value where you use a for expression to discard all of the null values immediately. The result would need to be an object rather than a map, because not all of the elements would necessarily have the same type, but for your purposes here I don’t think that matters:

locals {
  cells_definition = {
    for region, arns in var.cells_definition : region => {
      for service, arn in arns : service => arn
      if arn != null
    }
  }
}

Then you can use local.cells_definition instead of var.cells_definition in all of the places where you want to not see the attributes that map to null. This value would have a dynamically-chosen object type instead of being a map of objects as with the input, but as I say I don’t think it really matters for anything you intend to do with this value.

Another way to to look at it is to observe that Terraform’s type kind for a mapping whose keys are chosen dynamically, rather than fixed as part of the type, is maps, not objects. If you define this as map(map(string)) instead then that would potentially allow you to also validate for keys you aren’t expecting, rather than Terraform silently discarding them during object type conversion:

variable "cells_definition" {
  type = map(map(string))

  validation {
    condition = alltrue(flatten([
      for arns in var.cells_definition : [
        for service, arn in arns : contains(["elasticloadbalancing", "autoscaling", "lambda"], service)
      ]
    ]))
    error_message = "Supported service names are autoscaling, elasticloadbalancing, and lambda."
  }
}

The above would allow any mapping with a subset of those three keys, but reject any keys not included in the set of valid keys.

4

I really like the 2nd idea you proposed. It made it easier to increase the amount of services we’re supporting in our module. I also added validation for “like a region” for the key :

  type = map(map(string))
  validation {
    condition = alltrue([for _, k in keys(var.cells_definition): can(regex("[a-z][a-z]-[a-z]+-[1-9]", k))]) && alltrue(flatten([
      for arns in var.cells_definition : [
        for service, arn in arns : contains(["elasticloadbalancing", "autoscaling", "lambda", "apigateway", "kafka", "rds", "ec2", "route53", "sns", "sqs"], service)
      ]
    ]))
    error_message = "Supported service names are elasticloadbalancing, autoscaling, lambda, apigateway, kafka, rds, ec2, route53, sns, or sqs."
  }

Thanks a ton, Martin!