Variables in provider config

[lijok]

Hi

The docs over at https://www.terraform.io/docs/configuration/providers.html state:
You can use expressions in the values of these configuration arguments, but can only reference values that are known before the configuration is applied. This means you can safely reference input variables, but not attributes exported by resources (with an exception for resource arguments that are specified directly in the configuration).

Which would make me understand that a config like this:

provider aws {
  version = "~> 3.1"
  region  = "us-east-1"
  profile = "development"
}

provider grafana {
  url  = "https://grafana.mydomain.com"
  auth = aws_s3_bucket.this.arn
}

resource aws_s3_bucket this {
  bucket = "my-test-bucket"
}

would not work, because the grafana provider config is referencing an attribute exported by a resource

However, it does work, which raises the question of, whether the docs are incorrect, or are there special cases where a setup like this can work?

[apparentlymart]

Hi @lijok,

The effect of what you wrote here is that if you do an initial terraform plan, before any of these objects have been created, Terraform will tell the Grafana provider that the auth argument is “unknown” during planning. For many providers, that would lead to an error because they need to know their configuration in order to fully initialize.

I think the reason this particular situation happened to work is because the grafana provider in particular just stashes away the values it gets during configure without doing any pre-validation on them at all, and so it just saves that unknown auth value and awaits further instructions. As long as nothing else Terraform asks the provider to do during planning depends on that auth argument, it can succeed by virtue of just ignoring the “invalid” value altogether.

The documentation you referred to is describing this in the general case from the perspective of the core Terraform language, but ultimately the handling of this situation is down to individual providers, and so some of them either accidentally make this work (as we see here) or are intentionally designed to support a subset of their operations in a partially-configured mode.

Both of these are relatively rare, so the language documentation is erring on the side of describing the behavior you can rely on – that known values in the configuration will always work – and leaving individual providers to document their exceptions to that rule, where relevant. (I imagine the Grafana provider does not because it seems more like a happy accident of how it’s implemented than of intentional design.)

[lijok]

Thanks @apparentlymart, that makes sense

Looking at the graph produced by terraform graph | dot -Tsvg > graph.svg, which I attached, am I understanding correctly that the dag is interpreting the dependencies of this config correctly, so it would be safe for me to use this kind of configuration?

EDIT: To clarify, I’m talking about using this type of config, not the config posted specifically. The reason I need the provider to be configured with resource outputs is to overcome the chicken and egg problem, since we’re deploying grafana with terraform, which we then use to issue the api key, which in turn is used in the provider config, so that we can then go and use grafana resources to create dashboards, all in one apply

[apparentlymart]

Hi @lijok,

A subtlety to know about if you are interpreting the graph output is that this only reflects the sequence of actions for a particular sub-operation (by default, as in this case, the creation of a plan). It doesn’t represent the full set of actions required to create a plan and apply it, because applying is an entirely separate sub-operation with its own graph.

The upshot of this for your example here is that Terraform will indeed plan the necessary actions for the aws_s3_bucket.this resource before it configures provider.grafana, but if the result of planning the bucket is to produce a create action (that is, if there is not yet a remote object associated with the resource), the valuation of that bucket might be something like this:

  bucket = "my-test-bucket"
  arn    = (known after apply)
  # (and all of the other arguments/attributes)

Terraform will then visit provider.grafana and evaluate its configuration, producing an object like this:

  url  = "https://grafana.mydomain.com"
  auth = (known after apply)

That value – including the “known after apply” placeholder (which is how I’m representing the idea of “unknown” for the sake of this explanation) – is what Terraform Core sends to the provider as its configuration. The provider ultimately decides how to react to that unknown value; the grafana provider happens to accept it, as I described in my previous comment.

Terraform will then move on to grafana_folder.this and ask the provider to produce a plan for that object. Again, it’s entirely up to the provider how to respond to that request, and in this case the provider seems to succeed.

There were two opportunities in this path where things could’ve failed, though:

• The call to configure the grafana provider in the provider.grafana node might’ve rejected the unknown value outright.
• The call to plan grafana_folder.this in the grafana_folder.this node might’ve failed if it were necessary to contact the Grafana server to produce an accurate plan for that object, because no auth value is available.

---

For the sake of completeness, there is also one other possible path that can potentially allow things to succeed: the AWS provider could return a known value for arn as part of creating a plan for aws_s3_bucket.this. Exported attributes are typically values that are determined by the remote API rather than the provider itself, but in situations where a provider has enough information to predict a specific result it can include that known value as part of the plan. In this case, the AWS provider could potentially apply the rule that the bucket ARN is just a fixed prefix on the bucket name and return the ARN that would result from that during planning:

  bucket = "my-test-bucket"
  arn    = "arn:aws:s3:::my-test-bucket"
  # (and all of the other arguments/attributes)

As far as I know the AWS provider does not do this, but if it did then the arn would be known during the plan phase and so the Grafana provider configuration would be entirely known:

  url  = "https://grafana.mydomain.com"
  auth = "arn:aws:s3:::my-test-bucket"

Again, this is something that only works when intentionally supported by a provider, so it’s not something we can generally rely on and so the Terraform Language documentation doesn’t talk about it. If the AWS provider were providing a bucket ARN during planning then it would presumably mention that in the documentation for that attribute, thus giving you the license to ignore the general guidance in the Terraform Language documentation when working with that attribute in particular.

(In practice, this sort of pre-population of values during planning is not commonly implemented, because provider developers try to avoid re-implementing logic implemented by the remote APIs in the provider in case that logic changes later. However, it is a capability available to provider developers in situations where the benefit of doing so outweighs the potential risk of drift in the upstream API design.)

[lijok]

Thank you, that’s very insightful
I think I’ll be able to run with that then to solve this issue

Thanks again