Request for Feedback: Optional object type attributes with defaults in v1.3 alpha

An important distinction to be made here is that an explicitly set null value can only be determined at the syntax level, not within the language in general. This means when assigning nested objects to types with defaults, nested unset attributes will inherently have a null value and therefore we need to decide whether those will get a default or not. The choice is somewhat arbitrary, but a choice needs to be made nonetheless, and it’s consistent with the definition of the Terraform language to treat the null attribute as unset.

In comparison to the nullable = true aspect of a module variable, that was more a mistake than anything, and only applies to the topmost level of a module variable, not the attribute nested with the object types.

You both keep calling it a “mistake.” I think I’d call it something closer to serendipity. People discover great things all the time by mistake, or by accident, or through failure. I’ve written hundreds of modules, and probably thousands of input variables. I find myself very often taking advantage of the nullable = true aspect of input variables. I have only twice decided that nullable = false was more appropriate for an input of the module.

So because being nullable was a “mistake,” and because of backwards compatibility promises, but since is this technically a new “feature” and so it is not subject to that promise, now we’re going to evaluate inputs for nested object types differently than inputs for variables.

All of these options will support nullable inputs:

  • Variable with no default (required)
  • Variable with a null default (optional)
  • Variable with a non-null default (optional)
  • Nested object attribute with no default (required)
  • Nested object attribute with a null default (optional)

But this option for some reason will be different and won’t be nullable:

  • Nested object attribute with a non-null default (optional)

That just feels inconsistent. And seems likely to cause more confusion for users.

I don’t know. I guess I just don’t really see users making the distinction when they write their input values that these two things might work differently, just because one is nested and one is not.

But it’s fine I guess. I will of course figure out something that works for us, either way.

Hi - thanks for giving us Terraform!!!
I posted this into a discussion thread on GitHub, but am adding it here too…

I would suggest that (though the v1.3.0 adding of the 2nd default field in the “optional()” function is an improvement) the “optional()” keyword/function itself is really a bit of a red-herring and simply expanding the variable structure to allow nesting would be a more natural and syntactically consistent approach.

The code would look something like this…

A normal simple variable declaration:

variable "name" {
  type        = string
  description = "The name"
  default     = "blah"
}

A complex variable declaration:

variable "settings" {
  description = "The settings for my widget"
  type        = object({
    name          = string       # shorthand for "{ type = string }"
    notes         = { type = string, default = "Many things to do" }
    how_many      = { type = number, default = 1, description = "Count of items" }
    list_of_items = { type = list, default = [] }
  })
}

Just as with simple variables, the act of declaring a default makes it optional; there is no need for an “optional()” keyword/function and conversely not specifying a “default” makes it a required item.

Note that it also allows a description for each item in the object.

Thinking about it, the “object()” declaration could become redundant too, as the fact that there is a mix of types would implicitly make that so.

variable "settings" {
  description = "The settings for my widget"
  type        = {
    name          = string       # shorthand for "{ type = string }"
    notes         = { type = string, default = "Many things to do" }
    how_many      = { type = number, default = 1, description = "Count of items" }
    list_of_items = { type = list, default = [] }
  }
}

Extrapolating that further, the “type =” lines could be recursive and thus support sub-objects .

Hi @paul.rowlands,

Something like what you suggested here is indeed what I was imagining when I said it should be possible to extend the syntax in a backward-compatible way in future if the need arises based on practical experience.

To make such a large design change at this point would likely cause this entire feature to be postponed to v1.4 so that there’s time to evaluate the various new questions that raises, and there is so much pent-up demand for making optional attributes work that I don’t think that would be favorable in the community, so I think the most pragmatic option is to move forward with this small extension to the already-well-tested optional attributes experiment and then learn from how people use it what other features might be important and then, once we know what those are, to design a suitable syntax to express them.

Thanks @apparentlymart
Yes, I think there is a pressing need for the ability to handle defaults vs. the need to be artistic.
I’ll take “optional()” with the 2nd default arg asap, please!
(I’ve just spent a load of time working around the issue and making my code look ugly in the process.)
Is it still going to be left as an experimental feature?
I would’ve thought (in my naive world) that the underlying functionality would be pretty much there, with the only (ha!) requirement being to parse the source code accordingly.
If that can be done in 1.4, then that’d be great!

FYI, pretty much this was suggested in a proposal in 2019.

https://github.com/hashicorp/terraform/issues/22449

1 Like

Our intention is to stabilize the current behavior in v1.3 unless we hear some feedback that’s strong enough to justify holding back, which at this point is quite a high bar to hit because of how much feedback we’ve already heard about folks successfully using the current design and the experimental version that preceded it.

Yes, fully agree and I’m not pushing (or even suggesting, nor even merely hinting!) that this should be in the imminent 1.3 release; please don’t let me divert attention from that.
It’d be good to see the more elegant proposal implemented in a future release (1.4 if possible, or further up the road if needs be).
Looking forward to 1.3!

This is absolutely awesome.

  • Any idea when 1.3 will be released, are we talking a few weeks, a few months?
  • Any chance that the deep-merge functionality that this provides could be made available additionally in a function deepmerge() that could be called in resources etc, since as discussed in a github issue, this is currently missed by many

The problem with “deep merge” is still that there are several different competing definitions of what it ought to do. None of them seem to match what the optional attributes defaults mechanism does (the closest to that was the defaults function from the experimental optional attributes, but many folks found what that did counter-intuitive compared to what they imagined deep merge might do.

One day I would like to include a function that can convert a value to a particular type constraint using the same syntax that we use for variables, but when we attempted that we found ourselves blocked by the fact that type constraints involve some special symbols that have a different meaning in normal expressions: string, any, etc are understood by Terraform as potentially being resource type names in the main expression language. So far it isn’t clear how to get there without at least a small breaking change. If we could do it though, that would expose the defaults merging behavior in particular for use in expressions.

Hi all! Thanks for your feedback here.

This topic was helpful for gathering less structured feedback during the development process, but now that we’re in the prerelease period it’ll be better to share any bug reports and feature requests via GitHub issues in the usual way, since then we can make sure they get tracked appropriately and prioritized.

For that reason, I’m going to close this topic now. If you have found a problem with this feature in the beta1 release or later, please open a GitHub issue! Thanks again.