Request for Feedback: Optional object type attributes with defaults in v1.3 alpha

Terraform
1

Hi all :blush:,

I’m the Product Manager for Terraform Core, and we’re excited to share our v1.3 alpha , which includes the ability to mark object type attributes as optional, as well as set default values (draft documentation here). With the delivery of this much requested language feature, we will conclude the existing experiment with an improved design. Below you can find some background information about this language feature, or you can read on to see how to participate in the alpha and provide feedback.

To mark an attribute as optional, use the additional optional(...) modifier around its type declaration:


variable "with_optional_attribute" {
  type = object({
    a = string                # a required attribute
    b = optional(string)      # an optional attribute
    c = optional(number, 127) # an optional attribute with default value
  })
}

Background

As many of you know, in Terraform v0.14 we introduced experimental support for an optional modifier for object attributes, which replaced missing attributes with null. Your helpful feedback validated the idea itself, but highlighted a need for specifying defaults for optional attributes. In Terraform v0.15, we added an experimental defaults() function, which allows filling null attribute values with defaults. This resulted in extensive community feedback, and many of you found the defaults function confusing when used with complex structures, or inappropriate for your needs.

We know this experiment has been out in the wild for some time, and we’re incredibly grateful for your patience and feedback on the necessity of this language feature. With that, we’d love for you to try the new syntax available in the v1.3 alpha, and provide any and all feedback.

How to provide feedback

This feature is currently experimental, with the goal of stabilizing a finished feature in the v1.3.0 release. That being said, your feedback and bug reports are vital to us confidently releasing this feature as non-experimental during this release cycle.

Experience reports

Please try out this feature in the alpha release, and let us know what you think. For example:

  • Does this new design solve your problems?
  • Do you have any feedback on the semantics?
  • Is the documentation sufficiently clear?

Bug reports

  • For any bugs, please open issues in our repository here.
  • For any edge cases that are not solved by this design, you can also open an issue in our repo.

General feedback
For general feedback, please comment directly on this post. If you’d prefer to have a private discussion, you can email me directly (kalpers@hashicorp.com). However, public posts are most helpful for our team to review.

Note

Because this feature is currently experimental, it requires an explicit opt-in on a per-module basis. To use it, write a terraform block with the experiments argument set as follows:

terraform {
  experiments = [module_variable_optional_attrs]
}

Until the experiment is concluded, the behavior of this feature may see breaking changes even in minor releases. We recommend using this feature only in prerelease versions of modules as long as it remains experimental.

Thank you again for all your contributions to Terraform, and we’re so excited for you to try this release.

14 Likes
2

Hi,

I have been using this feature since Aug last year since I couldn’t find a better way to address my use case. I have been subscribed to the PR where this feature was being tracked since then and as soon as I got the notification from you @korinne, I changed the implementation I have before to support the new changes you announced. I love the fact that now I can set a default value much easier than I was doing before.
I will be impatiently waiting for this new 1.3.0 release

3 Likes
3

Awesome, thank you for trying the alpha out, and I’m so glad the feature meets your use-case. We are very much excited to release v1.3.0 as well :smiley:

4

I have been using this feature in my modules and using coalesce to define default values. This new syntax is much better to work with. Looking forward to its release.

1 Like
5

That’s great to hear, thank you!

6

Hi @korinne,

Just trying this out - thank you for the long awaited feature :slight_smile:

We use a lot of nested objects in the module I help maintain. I know this is complex…

In the current implementation, any nested objects have to be defined, even if all of their contained properties are also optional.

E.g.

variable "nested_objects" {
  type = object({
    enabled = optional(bool, true)
    settings = object({
      setting1 = optional(string, "")
      setting2 = optional(number, 0)
    })
  })
  default = {
    settings = {} # don't want to have to specify this, as all properties are optional
  }
}

In this scenario, I’d like Terraform to infer that, since var.nested_object.settings.setting[1,2] are optional, that I do not need to specify settings key in the default.

Ideally this would work with multiple levels of nested object too.

Is this possible?

3 Likes
7

If I understand your goals correctly, then yes, this is possible! You can make the settings attribute optional, and specify an empty object {} as its default. Terraform will then apply the specified default attributes inside settings.

Consider this configuration:

terraform {
  experiments = [module_variable_optional_attrs]
}
variable "nested_objects" {
  type = object({
    enabled = optional(bool, true)

    # settings is optional, defaults to object
    # with all default values applied
    settings = optional(object({
      setting1 = optional(string, "")
      setting2 = optional(number, 0)
    }), {})
  })

  default = {} # no need to include settings here
}

output "nested_objects" {
  value = var.nested_objects
}

Without specifying a variable value:

$ terraform plan

Changes to Outputs:
  + nested_objects = {
      + enabled  = true
      + settings = {
          + setting1 = ""
          + setting2 = 0
        }
    }

Only specifying the enabled attribute:

$ terraform plan -var nested_objects="{ enabled = false }"

Changes to Outputs:
  + nested_objects = {
      + enabled  = false
      + settings = {
          + setting1 = ""
          + setting2 = 0
        }
    }

Overriding a settings default:

$ terraform plan -var nested_objects="{ settings = { setting2 = 5 } }"

Changes to Outputs:
  + nested_objects = {
      + enabled  = true
      + settings = {
          + setting1 = ""
          + setting2 = 5
        }
    }

Does this work with your use case?

4 Likes
8

This looks perfect! Thanks, will give it a go :+1:

1 Like
9

@alisdair Just wanted to say thanks again - works a treat!

2 Likes
10

A quick update on this feature: the most recent alpha build has concluded the experiment, so the terraform { experiments = … } setting should be removed.

Feedback on this change is still very much welcomed during the 1.3 prerelease phase!

3 Likes
11

@alisdair @korinne Does it work with

type = map(object({
)}

?

For example, Can I do this?

variable "node_pools" {
  type = map(object({
    name                 = string
    cluster_name         = optional(string, "k8s-private")
    location             = optional(string, "us-central1-a")
    node_config          = object({
      machine_type       = string
      image_type         = string
      labels             = map(string)
      oauth_scopes       = list(string)
      taint              = list(map(string))
      tags               = list(string)
    })
    autoscaling          = object({
      min_node_count     = number
      max_node_count     = number
    })
    management           = object({
      auto_repair        = bool
      auto_upgrade       = bool
    })
    upgrade_settings     = object({
      max_surge          = number
      max_unavailable    = number
    })
  }))
}
12

Yes, that’s a valid variable type constraint. You can use optional(<type>, <default>) on any attribute of any object at any level of a type constraint.

I recommend download the latest alpha build and trying it out to check if the behaviour of the defaults is suitable for your use case.

14

I just pulled in the latest 1.3 Alpha, and made the necessary changes (removed defaults() and replaced with the new concise default declaration syntax, and removed the experiments setting) and I love it!

I’ve been using this feature for a long time now, and being able to set the defaults in my variable declarations, rather than later in locals, makes this code much more easy for my team to read and operate.

Thank you!

2 Likes
15

That’s great to hear, thank you!

16

Hi, I just found this as I was looking through setting defaults for an object used to define EKS Jobs in a Step Function module. Is it possible to set defaults for nested objects (like two levels in)?

variable "jobs" {
  description = "Details on the EKS jobs to be included in the workflow"
  type = list(object({
    name = string
    containers = list(object({
      name    = string
      image   = string
      command = optional(list(string))
      args    = optional(list(string))
      env     = optional(list(map(string)))
      resources = optional(object({
        limits = optional(object({
          memory = optional(string, "16Mi")
          cpu    = optional(string, "100m")
        }))
        requests = optional(object({
          memory = optional(string, "16Mi")
          cpu    = optional(string, "100m")
        }))
      }))
      securityContext = optional(object({
        runAsUser = optional(number, 1000)
      }))
    }))
    end = optional(bool)
  }))
  default = [
    {
      name = "Hello1"
      containers = [{
        "name"  = "busybox",
        "image" = "busybox",
        "command" = [
          "echo"
        ],
        "args" = [
          "Hello from Step!"
        ],
        "env" = [
          { name = "test_env1", value = "test_val" }
        ],
        }
      ]
    }
  ]
}

In the above, var.jobs[0].containers.resources still comes out as null. Security context is fine however, resources remains null. See below from terraform console test:

tolist([
  {
    "args" = tolist([
      "Hello from Step!",
    ])
    "command" = tolist([
      "echo",
    ])
    "env" = tolist([
      tomap({
        "name" = "test_env1"
        "value" = "test_val"
      }),
    ])
    "image" = "busybox"
    "name" = "busybox"
    "resources" = null /* object */
    "securityContext" = {
      "runAsUser" = 1000
    }
  },
])
17

Actually, I think I figured it out and the following configuration works. The feature looks cool. Any idea when 1.3.0 would be ready? Going to add this alpha version to our terraform cloud organization for now

variable "jobs" {
  description = "Details on the EKS jobs to be included in the workflow"
  type = list(object({
    name = string
    containers = list(object({
      name    = string
      image   = string
      command = optional(list(string))
      args    = optional(list(string))
      env     = optional(list(map(string)))
      resources = optional(object({
        limits = optional(object({
           memory = optional(string, "16Mi")
           cpu = optional(string, "100m")
        }))
        requests = optional(object({
           memory = optional(string, "16Mi")
           cpu = optional(string, "100m")
        }))
      }), {
        limits = {
          memory = "16Mi",
          cpu = "16Mi"
        }
        requests = {
          memory = "16Mi",
          cpu = "16Mi"
        }
      })
      securityContext = optional(object({
        runAsUser = optional(number, 1000)
      }))
    }))
    end = optional(bool)
  }))
  default = [
    {
      name = "Hello1"
      containers = [{
        "name"  = "busybox",
        "image" = "busybox",
        "command" = [
          "echo"
        ],
        "args" = [
          "Hello from Step!"
        ],
        "env" = [
          { name = "test_env1", value = "test_val" }
        ],
        }
      ]
    }
  ]
}
18

Hi @EmmanuelOgiji,

It looks like you figured out that you can set a default value for the object as a whole in order to make it appear as a default object when omitted, rather than as null.

An extra note on top of that is to notice that Terraform will automatically convert the default value you specify to the type constraint you specified, and so if your nested object has optional attributes with defaults itself you do not need to duplicate them in the default value of the object, and can instead provide an empty object as the default and have Terraform insert the default attribute values automatically as part of that conversion:

      resources = optional(object({
        limits = optional(object({
           memory = optional(string, "16Mi")
           cpu = optional(string, "100m")
        }))
        requests = optional(object({
           memory = optional(string, "16Mi")
           cpu = optional(string, "100m")
        }))
      }), {})

Notice in the above that the default value specified for resources is {}. Terraform will try to convert that to the specified object type, and so in the process it will have the default values of limits and requests inserted into it in the same way that a caller-specified empty object would, and so omitting resources altogether would produce the same final default value as the one you wrote out manually in your example, without the need to duplicate those defaults.

For completeness and for anyone else who is reading who might have a different situation: note that {} was only a valid default value here because all of the attributes in the resources object type are marked as optional. If there were any required attributes in there then the default value would need to include them in order to make the default value convertible to the attribute’s type constraint.

19

Beautiful, I have changed my code and so far so good. I’ll report any issue :wink:

20

Hi everyone,

I’ve created an issue but it seems my point is more relevant here: Release of optional attributes in TF 1.3 breaks modules using the experimental feature even if compatible · Issue #31355 · hashicorp/terraform · GitHub

TL;DR: IMHO, Terraform 1.3 should introduce a warning instead of an error when experiment is enabled within a module.

Context: we maintain around 70 modules and have activated the experimental feature in a bunch of them lately.
We have CI on all of them and on of the test is that module is compatible with latest providers and Terraform version. CI has began to break for latest Terraform aplha version.
We fully agreed that the feature was experimental and could break at any time … but we did it despites this, and I think we’re not the only ones.

In Terraform 1.3, having the experiment enabled in a module prevent the user to init its stack, and so, prevent the user to be able to use Terraform 1.3 although the syntax is almost the same (except for managing the default value, which is by the way a lot better in new implementation).
My point is that Terraform should display a warning message when encountering the experiment flag instead of breaking with an error since modules using it are mostly compatible.

Also, thanks for releasing this very handy feature :+1:

Thanks

21

Hi @BzSpi,

I replied in the GitHub issue before seeing your feedback here and so I won’t repeat all of what I said over there, but I will restate the most important part: any module using new features introduced in a particular Terraform version will always require using that version, and so as usual a module which uses optional attributes will inherently require using Terraform v1.3 or later because that is the first version that truly supported the feature.

It is interesting that in this particular case there is some overlap between the experimental design and the final design, but experimental features are not part of the language and any module using them should expect to become “broken” either by future iterations of the experiment or by the experiment concluding and that experimental functionality therefore no longer being needed for its intended purpose, which is requesting early feedback in discussion threads such as this one. In recognition of that not having been clear in the past, we are planning to make experiments available only in alpha releases in the future, with stable releases only supporting the stable language.

1 Like