Is it realistic to expect that Terraform could apply the whole state at once?

Terraform
1

Hi,

is it realistic to expect that Terraform could apply the whole state at once? For example, we have GCloud infra and provision kube cluster. On top of that we also need to provision namespaces inside the cluster so the cluster needs to be up & running & in a healthy state before the namespace can be provisioned. However, applying the namespace fails to “connection refused” - likely due to kube cluster still being in somewhat unhealthy state. This can be worked around by trying to apply the state more than once.

So the question is: Is it realistic to expect that Terraform could really be applied at once in a real world project and expect a single terraform apply to be enough? Or is it business as usual that some resource dependencies requires the apply to be run more than once?

E: Asked the same question in Stack Overflow: Is it realistic to expect that Terraform could apply the whole state at once? - Stack Overflow

2

have you looked at depends_on to make the namespace wait for the cluster. It may not work if the API call is synchronous and there is then stuff going on in GCP that Terraform doesn’t know about (e.g. getting to a ready state) but I’ve used it on AWS to wait for database clusters to create before doing other configuration.

3

It sounds like you have a situation where the configuration for one provider refers to an object that’s not yet created by some other provider. In your case, I assume it’s the hashicorp/kubernetes provider configuration referring to something from the hashicorp/google or hashicorp/google-beta providers.

In that case indeed it’s not typically possible to configure across multiple systems in a single Terraform configuration, because Terraform needs to configure a provider in order to plan against that provider, but at the first plan the hashicorp/kubernetes provider configuration is incomplete. There’s more about this in the Provider Configuration documentation:

You can use expressions in the values of these configuration arguments, but can only reference values that are known before the configuration is applied. This means you can safely reference input variables, but not attributes exported by resources (with an exception for resource arguments that are specified directly in the configuration).

How a provider reacts to an incomplete configuration is up to that provider, but IIRC the Kubernetes provider in particular reacts to it by trying to connect to some default location, such as localhost, and thus you see “Connection Refused” if you are running Terraform on a system where Kubernetes isn’t running.

The most robust answer to this problem is to split into two configurations where one is responsible for establishing the underlying cluster and then another is responsible for configuring that cluster.

However, since this is only a bootstrapping problem (that is, it goes away once you’ve created everything for the first time and move into a mode of just updating these objects), this can also be a valid situation to use Resource Targeting, allowing you to perform an initial run to establish the cluster and then finally converge on a complete system by running an untargeted apply:

# first create just the cluster and its prerequisites:
terraform apply -target=google_container_cluster.example
# then, after that succeeds, apply everything to converge:
terraform apply

After this initial bootstrapping process you can then just use normal terraform apply on an ongoing basis as long as you don’t destroy or replace the container cluster.

4

Thank you. This was exactly what was happening! I think this is a pitfall most of the terraform users hit in the beginning and it is not clear without extensive prior knowledge what is actually happening and why - terraform does not give any hint that the variables might not be initialized at this point. I was not aware of the -target variable, looks promising!