Terraform Errors and How to Debug Them

Terraform Providers Kubernetes
1

Hi everyone,

not sure what I should call the topic of this message. The error message is specific but I wonder how you guys are coping with such errors and how to analyze them. So if you haven’t seen this* very specific error, but can share some details on how to cope with such errors in general, this would be very appreciated!

Some background info: I think the errors origins from a helm provider upgrade and applying some changes, then reverting back the helm provider and now a terraform plan spits outs:

╷
│ Error: Failed to transform List value into Tuple of different length
│
│   with module.dev-k8s.kubernetes_manifest.cert_manager_issuer,
│   on modules/kubernetes/certmanager.tf line 55, in resource "kubernetes_manifest" "cert_manager_issuer":
│   55: resource "kubernetes_manifest" "cert_manager_issuer" {
│
│ Error: %!s(<nil>)
│ ...at attribute:
│ spec.acme.solvers

Line 55 is the first line below:

resource "kubernetes_manifest" "cert_manager_issuer" {
  manifest = {
    apiVersion = "cert-manager.io/v1"
    kind       = "Issuer"
    metadata = {
      name      = "letsencrypt-prod"
      namespace = kubernetes_namespace.cert_manager.metadata[0].name
    }
    spec = {
      acme = {
        email  = "foo@bar.com"
        server = "https://acme-v02.api.letsencrypt.org/directory"
        privateKeySecretRef = {
          name = "some-key"
        }
        solvers = [
          {
            dns01 = {
              cloudDNS = {
                ...
              }
            }...
          }
        ]
      }
    }
  }
}
2

Hi @hashicorp17!

Unfortunately I don’t think there’s a general answer to how to diagnose the cause for all errors. Many of Terraform’s error messages include specific suggestions about what might be wrong and/or how to proceed, but there are many different error situations throughout both Terraform Core and all of the providers and so it’s inevitable that some of them will be less complete than others, particularly if describing an unusual situation that the developers didn’t consider.

This error seems to be coming from the hashicorp/kubernetes provider itself. I determined that because it says with module.dev-k8s.kubernetes_manifest.cert_manager_issuer, which means that Terraform was working on an action against this specific resource instance when the error occurred. (I also happen to know this isn’t a Terraform Core error because I work on Terraform Core, but I understand that heuristic isn’t useful to anyone else but me.)

Because this is a Kubernetes provider error I’ve moved this into the category for the Kubernetes provider since I expect you’re more likely to find someone who is familiar with this class of error in that category.

You did ask for pointers on how to debug in general though, and you’re less likely to hear about that in the new category so in addition to what I shared above I’ll add some other notes about what I might do next:

  • When I encounter a provider error I’m unfamiliar with, I typically start by trying to find it in the provider’s own codebase. In this case, that’s hashicorp/terraform-provider-kubernetes, which you could find via the provider’s page in Terraform Registry.

  • In this case I took a guess that “Failed to transform” is probably a fixed, hard-coded string in the codebase even if the subsequent words might be inserted dynamically, and so I searched for those three words in the repository source code.

  • At the time I’m writing this, there’s an exact match for that error message in a file called manifest/morph/morph.go:

    terraform-provider-kubernetes/morph.go at 0a6d49bc8d01a0f99fca3ee508f7b2a27317c04f · hashicorp/terraform-provider-kubernetes · GitHub

  • This code is doing something I’m not super familiar with, but from context (I know what kubernetes_manifest is intended to achieve) I’m inferring that what it’s trying to do is convert your overall manifest value to match the schema required by this Kubernetes resource type, so that it can submit the manifest to the Kubernetes API.

  • I can see in the code which emits the error message that the “at attribute:” part of the message is showing the path within the manifest structure that the problem relates to. It refers to spec.acme.solvers, which in your example seems to be a tuple of objects.

  • This is where the trail runs dry for me based on what I know already: I can see that the expression you wrote is a tuple already, but the error message says it’s converting a list into a tuple. Therefore there seems to be some other prior step which already converted your tuple into a list, and now this is trying to convert it back to a tuple again.

    However, I can share one useful detail about Terraform’s type system: a “list” type represents a sequence with any number of elements that all have the same type, whereas a “tuple” type represents a fixed-length sequence where each element can have a different type. Terraform and Terraform providers often use tuples as a starting value and then convert from there to lists by first converting all of the tuple elements to be of the same type.

    I expect what’s going on here then is that some earlier code already converted the objects inside solvers to all be of the same type matching the Kubernetes resource schema, and then produced a list from that result. But then as a separate step the schema seems to require a specific number of elements or perhaps different types for each element and that step is what failed here.

I’m afraid I can’t walk all the way to a solution here because I’ve reached the end of my expertise: I don’t know what is the schema for this resource type’s spec.acme.solvers and so I can’t explain what is wrong with the value you assigned to that attribute. However, hopefully you do know what is expected for that particular attribute and can focus on understanding why your value doesn’t match the schema.

1 Like
3

Hi @apparentlymart

thanks a lot for sharing how you analyze such errors! We still haven’t figured out the cause of the error, but the way you walked me through the different steps and thoughts is excellent and really useful!! Thanks a lot for taking the time to write this down.

4

@hashicorp17 @apparentlymart this seems to be related to the relevant CRD setting

x-kubernetes-preserve-unknown-fields: true

which causes other issues with the kubernetes_manifest resource because the provider can’t correctly assign a type to something that is basically untyped.