Idempotent vs data source

[darkn3rd]

This has to be the most annoying painful part of Terraform is that the data sources are not idempotent. If you fetch state from the cloud, and that resource does not exist, then Terraform exits and fails.

It should return an empty object that we can test for and use with count variable to deploy/provision resources based on the return status, not just raise an exception.

Because of this, it is all to commonplace to get code in a state where you can NEVER destroy the resource because Terraform exits with non-zero. So you end up having to purge the resources and follow up purging the state.

When you have to consistently hand hold the tool to get it to work properly, then the platform is FAIL. I hope this can be fixed in the future, e.g. Terraform 2.0.

[stuart-c]

Terraform expects you to either be creating & managing a resource, or not creating it and just consuming that resource (via a data source).

Even if you did get an empty response from a data source if the underlying resource didn’t exist (which is possible for some data sources) that still wouldn’t help. You can’t conditionally create a resource based on a data source trying to fetch information about the same resource - if it returned null and you conditionally created the resource on the next run it would return some data and you’d therefore remove the resource.

[darkn3rd]

If Terraform expects you to ONLY create a resource, then why have a data source concept in the first place. This is a lookup, the platform supports lookups.

There uses cases where you may want to say “if resource doesn’t exist, create it”. You cannot do this in Terraform, but you can on other platforms.

There’s also scenarios, where resources are created indirectly, such as Kubernetes, but if the process failed mid-way, you cannot delete the resources that were created, because the whole module will now fail. Thus you have to go around Terraform to fix this scenario, such as terraform state rm after removing the resource by another means. Another pattern, is to put guard rails, count, in every data source that can be overridden, so that you can manually tell Terraform don’t do the lookup, so that you can delete the resource.

[stuart-c]

There are two possible situations:

1. You don’t want this root module to manage a particular resource (it might be managed manually, automatically by your cloud provider, using a different IaaC tool or just in a different Terraform root module) but you still need to fetch some details about it. This is where you’d use a data source. There are other options too - for static resources you could just hardcode the details (possibly via a shared module) or for resources managed within another Terraform root module you could couple the two using remote state,

2. You want Terraform to manage a resource. In this case you’d use a normal resource block. If the resource already exists (for example it was previously manually created) and you want Terraform to take over the management (and you’ve checked that nothing else will try to manage it) you would use terraform import to tell Terraform to start managing it. If other things in the same root module need details about that resource you’d just reference the resource - there would be no use of data sources at all.

An example of where a data source might be useful is within AWS wanting to get the details of the latest AMI which has a particular set of tags. Some other process is actually creating the AMI, so a Terraform resource block isn’t appropriate, and hard coding the value isn’t possible because it changes.

[stuart-c]

@darkn3rd

There’s also scenarios, where resources are created indirectly, such as Kubernetes, but if the process failed mid-way, you cannot delete the resources that were created, because the whole module will now fail.

I’m not quite sure what you mean here. Could you give an example? If Terraform isn’t managing some resources (because they are created by something else indirectly) it shouldn’t know anything about them, and therefore nothing needs to be done within Terraform if things fail. I’m not sure if it is the situation you are thinking about, but if Terraform is used to install a Helm chart and for some reason that fails it is up to Helm to handle any rollback/errors. Terraform would just know that the Helm deployment failed rather than anything about the individual Kubernetes resources that it might have (or not) created. If the Helm deploy did fail on a subsequent Terraform run Helm would be asked to deploy again and would figure out how to achieve that.

[c4milo]

I’m fully in agreement with @darkn3rd here, we had to disable state refreshes during destroy operations precisely due to this design issue in Terraform. Terraform is a workflow orchestration tool, similar to a workflow engine. We all know what happens when the tasks of our workflows are not idempotent, it renders the entire thing unreliable, and that’s precisely what Terraform with all its providers is, unreliable.

[darkn3rd]

@stuart-c Adding more on this about the use case.

• AWS Certificate Manager
• Kubernetes deployment + service of type LoadBalancer
  • uses result of ACM to populate ARN value in an annotation
• Data source to read final result of LB provisioned by Kubernetes service of type LoadBalancer
• Register DNS record in Route53 managed zone for the output of the data source.

If for whatever reason, the service is not provisioned because the pods failed to come to a healthy state, then the whole thing falls apart. You cannot delete the provisioned resources because of the circular dependency. The data source blocks deletion, so the human operator has to then purge content from terraform state as well as purge the provisioned/deployed resources that can now never be managed by Terraform with the same code.

This is completely unnecessary, excessively complex, and is only a requirement to what I would conclude is a design flaw or shortcoming in Terraform.

In contrast, if i used Ansible that uses the true state of the cloud, or other imperative tool, like kubectl/helm/aws-cli, I would easily avoid this issue. For this reason, Terraform is problematic for managing Kubernetes resources in this regard.

[stuart-c]

That sort of setup is problematic (assuming this is a single root module) as you are trying to read something from Kubernetes which is also being created during the same apply.

We don’t use Terraform to update the DNS for Kubernetes Ingresses and instead use the External DNS application which handles it for us (from within the cluster) - a similar defined state type system, more more closely aligned with changes within Kubernetes, instead of requiring an external process.

[macmiranda]

Hi @darkn3rd,

Let me start off by saying that I don’t think Terraform is the right tool for your use-case. In fact, everything you want to achieve is doable the “Kubernetes-way”, without writing a single line of HCL (see more about this at the end).

@darkn3rd

If for whatever reason, the service is not provisioned because the pods failed to come to a healthy state, then the whole thing falls apart.

First of all, a Kubernetes Service should not fail because the Pods didn’t come to a healthy state. A Service can exist without any Endpoints and Kubernetes should create the Load Balancer resource no matter what but maybe the Terraform provider expects a different response from the Kubernetes API, which may be part of the problem.

The biggest problem I see though is depending on a resource that’s not being created through Terraform itself. Even though the k8s Service is, the Load Balancer isn’t (it’s created by a k8s controller). There are tons of things that can go wrong in that scenario. Kubernetes can wait forever to reconcile a resource that doesn’t match its desired state. Terraform can’t, so it must make a decision.

@darkn3rd

It should return an empty object that we can test for and use with count variable to deploy/provision resources based on the return status, not just raise an exception.

I get your point that the data source could return an empty object. This is actually true for data sources that return lists (e.g. aws_subnets). But if you’re adding a data source that references a single object, it’s because you want to use it somewhere else. If that object doesn’t exist, there is no point in moving forward.

In the specific case of your ELB, let’s assume, it returned an empty object. When you try to reference its dns_name or id in the aws_route53_record resource, it will raise an exception since either records or alias must be specified. What do you do then? Even if you add some sorta condition to work around this, you would still need to re-run terraform apply. Keep reading.

@darkn3rd

You cannot delete the provisioned resources because of the circular dependency.

Why should Terraform delete the resources it manages because of a failure it’s not capable of tracking? If the resources were already created, why would you need to destroy them? Your apply may be incomplete but a new plan will figure out the differences and create the missing resources (once you’ve resolved the problem with your external resources, not being managed by Terraform). I’m not excluding the possibility of a weird behavior of the Kubernetes provider but in general, you wouldn’t need to recreate failed resources (because they wouldn’t have been added to the state in the first place - if that’s your case, maybe you should raise a bug on the provider’s Github repo).

Depending on what you have to do to fix the problem that occurred outside of Terraform’s control, you may need to reimport the resources it manages, which brings me to my next topic.

@darkn3rd

There uses cases where you may want to say “if resource doesn’t exist, create it”. You cannot do this in Terraform, but you can on other platforms.

It’s true that Terraform has some limitations in that regard because it uses HCL which is a configuration language, not a procedural one, and it does assume a few things:

• If the resource exists, it should either be read as a data source or be imported to the state as a managed resource
• If it doesn’t, it will be created and managed by it
• Once managed, a resource should not be modified outside of Terraform because Terraform will attempt to return it to its desired state.

Some of these limitations could be easily overcome by using the Terraform CDK. That does not mean that Terraform as a platform is a “FAIL” though.

=== Alternatives ===

Based on what you explained your use-case is, I’d probably use the following approach:

• ACM certificate with wildcard for all k8s Ingresses (e.g. “*.domain.com”)
• AWS Load Balancer controller as the Ingress Controller
  • it manages ALB listener rules, target groups, and ACM certificates for Ingresses automatically
• Kubernetes deployments via FluxCD or ArgoCD using Helm charts
  • charts include an Ingress template for services that need to be exposed
• ExternalDNS makes sure the Ingress hostnames are created in Route53 as CNAME records pointing to the ALBs DNS name.

[darkn3rd]

@stuart-c

We don’t use Terraform to update the DNS for Kubernetes Ingresses and instead use the External DNS application which handles it for us (from within the cluster) - a similar defined state type system, more more closely aligned with changes within Kubernetes, instead of requiring an external process.

I was thinking about this, but this poses some challenges for cross cloud scenarios. AKS and GKE pods will need to update records on Route53, which creates a security challenge. If only on EKS, IRSA (OIDC) can be used to grant access to only that pod ExternalDNS. For the other, an easy route is static credentials, but this is very dangerous. It would be nice to use WI (OIDC) from either AKS or GKE, but this requires writing a webhook, and I am not sure how to set all this up to trust relationship between OIDC on GKE/AKS to AWS.

[stuart-c]

We use IRSA to grant the External DNS system limited access to Route53. I’m afraid we don’t use GKE so I can’t advise on how that might work, but I’d have thought it isn’t something other people haven’t encountered before, so I’d suggest looking at the Kubernetes/GKE forums for advice.

[darkn3rd]

@stuart-c

but I’d have thought it isn’t something other people haven’t encountered before, so I’d suggest looking at the Kubernetes/GKE forums for advice.

It’s not something well known and very niche. With an external provider setting up a trust relationship, it is technically possible. I imagine some code is needed to refresh the token one, but I am not advanced as far as programming AuthZ/AuthN/etc security, so I am unsure about this. I was pointing out that the recommended external solution outside of Terraform has some challenges/trade-offs.

Terraform uses credentials of the operator to upsert DNS records on Route53, which is more secure as only authorized operator has read-write, as opposed to creating an account identity w privileges for an external service. Ultimately, though this is proposed because of baseline of functionality that Terraform that IMO should have existed in Terraform by design from the beginning, e.g. idempotent lookups of cloud resources.

Additionally, this is just one use case, there are others, so ExternalDNS would not work for non-DNS use cases.