Provider's Schema - how to distinguish attributes from arguments?

Terraform
1

Hey,

I am working on Scala code generation out of terraform provider’s schema. To be more precise I use the output of terraform providers schema -json to generate some Scala code.

I am parsing attributes in the Block Representation which is described here: https://www.terraform.io/docs/commands/providers/schema.html#block-representation. I am having hard time to understand the meaning of required, optional and computed fields. There are some descriptions of them but they do not explain how those fields interact with each other. I also read https://www.terraform.io/docs/extend/schemas/schema-methods.html and https://github.com/hashicorp/terraform/issues/21278 but it did not help.

In particular I am interested in the difference between (computed: true) and (computed: true, optional: true). As the docs is not clear in that regard I hoped that I would be able to reverse engineer that by comparing schema of e.g. AWS Provider and its docs. However I was not able to identify a simple rule. Here are some (in my opinion) contradictory entries:

a) aws_acm_certificate.arn - (computed: true). According to docs it only output (i.e. attribute), cannot be passed as parameter
b) aws_dynamodb_table.stream_view_type - (computed: true). According to docs it’s an argument
c) aws_acm_certificate.subject_alternative_names - (computed: true, optional: true). According to docs it’s an argument

My point is that a) and b) have the same schema representation but have different semantics. b) and c) have the same semantics but different schema representation.

Can I really rely on schema representation to distinguish between attributes and arguments?

2

You could if the schema was properly declared, but in this case it appears that the schema is not. As best I can tell,'optional is irrelevant when computed is set to true, since the latter means that the user cannot supply the attribute in their configuration.

If I had to guess I’d say that both the schema and the docs for that provider have errors in them and those errors are causing confusion for you.

1 Like
3

I think the best way to think about these is that computed is mostly independent of optional vs. required.

The flag computed means that the provider may choose an arbitrary value (as long as it has the correct type) for that attribute. If computed is not set, then only the configuration can choose a value and the provider will honor it.

The presence of either optional or required signals that the configuration can provide a value for the attribute. In the case of required, the configuration must provide a value for the attribute. In the case of optional, if the attribute is not set in configuration then its value is null. optional and required are mutually-exclusive because their definitions conflict with one another.

The trickiest case is the one where both optional and computed appear together. That combines together the two situations above, meaning that the attribute can be chosen either by the configuration or by the provider, with the configuration taking precedence. Or to phrase it in a more user-oriented way: the provider can choose a default value to use if and only if there is no value set in the configuration.

In the interests of completeness, I’ll note also that required and computed conflict with one another because (given that configuration always takes priority over a provider’s defaults) there is no situation where a default value can ever be selected for a required attribute.

Perhaps the above will be easier to follow as a table of the valid combinations:

computed optional required Meaning
✓ The value is always chosen by the provider.
✓ The value is chosen by the configuration, or null if no value is set there.
✓ The value is chosen by the configuration and must always be set there.
✓ ✓ The value is chosen by the configuration if set, or by the provider if not set.

Looking at the JSON format documentation I can see that the comment attached to computed is not complete, because it only discusses the case where computed is used alone and doesn’t mention how it interacts with optional.

Another thing I want to note here, since I can see the potential for confusion, is that the terraform providers schema -json output is describing the schema in the terms Terraform Core understands, while the Terraform SDK (what the “extend” section of the documentation is discussing) is at a higher level of abstraction due to its own built-in validation behaviors and so while the concepts do overlap slightly, the extend documentation includes some details that do not apply to the terraform providers schema -json output.

Finally, some older provider features currently get a pass for exactly honoring the above requirements as an allowance for historical compatibility. That should phase out in the coming months as support for Terraform 0.11 in new provider releases comes to an end and the SDK can therefore use more modern behaviors throughout, but as I write this there are some situations you may encounter where a provider behaves as if it had set computed to true even though it actually did not.

For use-cases involving configuration generation those compatibility exceptions are usually not a problem since the values chosen by the provider are not involved in code-generation use-cases anyway. I point it out just in case you have different goals in your work where it might be significant that the final result of a Terraform action can potentially differ from the configuration even though computed wasn’t set.

2 Likes
4

That table should be in the SDK docs :slight_smile:

5

@apparentlymart @kpfleming Great thanks for your exhaustive answers! I find the table especially useful. It’s a bit worrisome that providers seems not to follow it one to one but I think some minor discrepancies will be acceptable in my use case.

6

Looking at this again with fresh eyes and particularly noticing how you phrased the topic summary:

When it comes to attributes vs. arguments, we consider “arguments” to mean the subset of attributes that are settable in the configuration, which are the ones that have either optional or required set. The ones that have neither set are not arguments, but they are still attributes.

7

@apparentlymart Thank you, now it’s clear.

Thus, I think, we can state that optional and required say us something only about how attributes can be used as inputs. They don’t say anything about “optionality” of attributes in general.

To make it more concrete, I will provide an example.

  • aws_acm_certificate.arn is (computed: true). It’s never provided by configuration but always present as output
  • aws_acm_certificate.domain_validation_options is (computed: true). It’s never provided by configuration and sometimes present in output. According to docs: Only set if DNS-validation was used

That is problematic for my use case because I would like to distinguish between values “always present” and “sometimes present”. I can express the latter with optional type to signal to the user necessity of checking against null.

This is a conclusion rather than question, but sharing it as it might be interesting for other users stumbling upon this topic. I would be happy to be proven wrong as that would make my task feasible.

8

Hi @note!

Indeed, Terraform’s type system doesn’t have the notion of non-nullable types: as far as the type system is concerned, any value could potentially be null (unset). Although indeed the provider-level logic might ensure that in practice the value is never null, that information is not visible in the schema and thus the behavior is not checked by Terraform Core.

Unfortunately I don’t think there’s any elegant answer to this. Seems to be a typical example of the sorts of wrinkles we encounter when trying to get interop between languages with different type systems. :confounded:

With that said though, I’ve never written Scala so I don’t know what sorts of patterns are idiomatic. I would assume that this sort of problem would arise also when interacting with JSON-based APIs and similar situations where values are arriving from a foreign type system, and so maybe there’s some other popular system in the Scalar community that could give inspiration for an idiomatic-ish solution?

9

I think I overstated my point that my task became unfeasible - it’s just part of the whole task. Probably I will generate all values except of required as nullable. Such model by itself would be cumbersome to use but the code generator will be just a part of the solution I work on. A user who is aware of meaning of each field would be still able to provide more precise types.

So it’s not a dealbreaker to me, I was just hoping I could get very precise types in completely mechanical way but seems that it’s not possible.

@apparentlymart Thank you for all your detailed answers! :slight_smile: They were really helpful