How to skip resources during terraform destroy?

Terraform
1

It seems there is an issue with order of deletion, the aws-auth configmap used for accessing the cluster is getting destroyed before the deletion of other resources, even after that EBS volumes are left behind, need help on that also. Any option to ignore this particular resource during the terraform destroy?

I tried prevent_destroy as below to skip the resource during terraform destroy.

resource "kubernetes_config_map" "aws_auth" {
  metadata {
    name      = "aws-auth"
    namespace = "kube-system"
  }

  data = {
    mapRoles = yamlencode(local.map_roles)
    mapUsers = yamlencode(local.map_users)
  }
  lifecycle {
    prevent_destroy = true
  }
}

But got error as below.

â•·
│ Error: Instance cannot be destroyed
│ 
│   on aws-auth.tf line 20:
│   20: resource "kubernetes_config_map" "aws_auth" {
│ 
│ Resource kubernetes_config_map.aws_auth has lifecycle.prevent_destroy set,
│ but the plan calls for this resource to be destroyed. To avoid this error
│ and continue with the plan, either disable lifecycle.prevent_destroy or
│ reduce the scope of the plan using the -target flag.
╵
2

Hi @ukreddy-erwin,

One option here is to tell Terraform to “forget about” this object, which will sever the binding between the object in the remote system and this block in your configuration. The object will continue to exist but Terraform will no longer have any record of it, and so terraform destroy will not propose to delete it.

terraform state rm kubernetes_config_map.aws_auth

When doing this, beware that it’s a one-way operation: once Terraform has “forgotten” this object it becomes the same as any other object in the remote system that this Terraform configuration didn’t create, and so if you want to re-bind it to the same address you will need to import it again.

If this is something you’ll encounter routinely, then I would suggest splitting your configuration into two parts so that you can create, update, and destroy them separately. Typically everything in a single Terraform configuration should be objects that you will create all at once and destroy all at once. You can use Data Sources to declare dependencies on objects created outside of the current Terraform configuration, which are therefore not subject to being destroyed as part of the current configuration.

data "kubernetes_config_map" "example" {
  metadata {
    name      = "aws-auth"
    namespace = "kube-system"
  }
}

The above snippet uses the kubernetes_config_map data source to retrieve the data from the config map whose resource block you showed in your question. In any configuration with the above block you can use data.kubernetes_config_map.aws_auth.data to retrieve the data that is stored in that config map.

2 Likes
3

No there are multiple resources like this, can’t add individual commands like this. Setting it within terraform file is more feasible so that direct terraform destroy should skip those, is more feasible

Is there anyway to tell terraform to delete this at the end, atleast.

4

what I observed, we have statefulset which creates some pods and they have linked pvcs. Those uses a configmap for authorization, all are part of terraform files, kubernetes provider. When ran terraform destroy, statefulsets are deleted and that triggers the pods to delete but terraform deletes immediately the configmap and the pods are still in terminating state even though the statefulset is removed, which caused the authorization issue and creating issue with pvc. So, if there any ignore option, that would be more feasible. Can’t have a separate terraform project just for this one entry.

5

Whenever you have various Terraform resources that depend on others, and those dependencies are not visible to Terraform automatically because you are referencing values from one in the definition of the other, you should use depends_on configuration in your Terraform resources so that it can know the required order of operations.

Note that this expresses the order in which things should be created - so the statefulset would depend on the configmap. Terraform knows to process them in the opposite order when destroying.

6

but the statefulset is cleared immediately as it is already had that configmap. but the individual pods are having issue with pvc.