Terraform Apply Does Not Update State, Pending Changes Always Show Without Applying

Hello,

I’m experiencing an issue with Terraform where changes are not being applied despite running the terraform apply command and confirming with yes. The pending changes are not reflected in the infrastructure or the state, and Terraform consistently shows the same pending changes even after running terraform plan.

Problem Details:

When we run terraform apply, Terraform shows the changes pending and asks for confirmation with yes, but after confirming, no changes are applied.

After executing terraform refresh, Terraform continues to show the same pending changes in the subsequent terraform plan.

This is the plan/apply

# module.custom_eks.module.eks.data.aws_eks_addon_version.this["coredns"] will be read during apply
  # (depends on a resource or a module with changes pending)
 <= data "aws_eks_addon_version" "this" {
      + addon_name         = "coredns"
      + id                 = (known after apply)
      + kubernetes_version = "1.32"
      + most_recent        = true
      + region             = (known after apply)
      + version            = (known after apply)
    }

  # module.custom_eks.module.eks.data.aws_eks_addon_version.this["eks-pod-identity-agent"] will be read during apply
  # (depends on a resource or a module with changes pending)
 <= data "aws_eks_addon_version" "this" {
      + addon_name         = "eks-pod-identity-agent"
      + id                 = (known after apply)
      + kubernetes_version = "1.32"
      + most_recent        = true
      + region             = (known after apply)
      + version            = (known after apply)
    }

  # module.custom_eks.module.eks.data.aws_eks_addon_version.this["kube-proxy"] will be read during apply
  # (depends on a resource or a module with changes pending)
 <= data "aws_eks_addon_version" "this" {
      + addon_name         = "kube-proxy"
      + id                 = (known after apply)
      + kubernetes_version = "1.32"
      + most_recent        = true
      + region             = (known after apply)
      + version            = (known after apply)
    }

  # module.custom_eks.module.eks.data.tls_certificate.this[0] will be read during apply
  # (depends on a resource or a module with changes pending)
 <= data "tls_certificate" "this" {
      + certificates = (known after apply)
      + id           = (known after apply)
      + url          = "[[HIDDEN]]"
    }

  # module.custom_eks.module.eks.aws_eks_addon.this["coredns"] will be updated in-place
  ~ resource "aws_eks_addon" "this" {
      ~ addon_version               = "v1.11.4-eksbuild.24" -> (known after apply)
        id                          = "[[HIDDEN]]:coredns"
        tags                        = {}
        # (12 unchanged attributes hidden)

        # (1 unchanged block hidden)
    }

  # module.custom_eks.module.eks.aws_eks_addon.this["eks-pod-identity-agent"] will be updated in-place
  ~ resource "aws_eks_addon" "this" {
      ~ addon_version               = "v1.3.10-eksbuild.1" -> (known after apply)
        id                          = "[[HIDDEN]]:eks-pod-identity-agent"
        tags                        = {}
        # (12 unchanged attributes hidden)

        # (1 unchanged block hidden)
    }

  # module.custom_eks.module.eks.aws_eks_addon.this["kube-proxy"] will be updated in-place
  ~ resource "aws_eks_addon" "this" {
      ~ addon_version               = "v1.32.9-eksbuild.2" -> (known after apply)
        id                          = "[[HIDDEN]]:kube-proxy"
        tags                        = {}
        # (12 unchanged attributes hidden)

        # (1 unchanged block hidden)
    }

  # module.custom_eks.module.eks.aws_eks_cluster.this[0] will be updated in-place
  ~ resource "aws_eks_cluster" "this" {
        id                            = "[[HIDDEN]]"
        name                          = "[[HIDDEN]]"
        tags                          = {
            "[[HIDDEN]]" = "[[HIDDEN]]"
        }
        # (14 unchanged attributes hidden)

      ~ encryption_config {
            # (1 unchanged attribute hidden)

          ~ provider {
              ~ key_arn = "[[HIDDEN]]" -> "[[HIDDEN]]"
            }
        }

        # (5 unchanged blocks hidden)
    }

  # module.custom_eks.module.eks.aws_iam_openid_connect_provider.oidc_provider[0] will be updated in-place
  ~ resource "aws_iam_openid_connect_provider" "oidc_provider" {
        id              = "[[HIDDEN]]"
        tags            = {
            "Name" = "[[HIDDEN]]"
        }
      ~ thumbprint_list = [
          - "[[HIDDEN]]",
        ] -> (known after apply)
        # (4 unchanged attributes hidden)
    }

Plan: 0 to add, 5 to change, 0 to destroy.

Every time I hit apply → yes . Nothing is changed, and the next apply/plan the result is the same.

Steps Already Taken:

We ran terraform refresh to update Terraform’s local state with the latest values from the infrastructure.

We used terraform plan afterward, but it still shows the same pending changes.

We executed terraform state show to verify the configuration in the state, but we don’t see significant differences from the actual infrastructure.

We attempted to import the resource using terraform import, but Terraform is already managing the resource in the state, causing conflicts when trying to re-import it.

Terraform Version: 1.14.0
AWS Provider Version: 6.23.0
Operating System: Tried on MacOS and Ubuntu

Hi @velik.georgiev,

The only managed configuration change I see in your plan output is the key_arn attribute of the aws_eks_cluster, but the the masked change only says [[HIDDEN]]" -> "[[HIDDEN]]". Is there a difference between before and after, and is that final value saved in the state?

The rest of the changes are probably due to the data resources not being able to be read during the plan, I’m guessing because the data sources depend on the aws_eks_cluster.

If the aws_eks_cluster is not applying the correct value, you may be able to see something in the warning logs about that resource behaving incorrectly (it’s allowed certain data inconsistencies due to the AWS provider still using the legacy SDK). You might also confirm that using -refresh=false, which would then show no changes if the resource was otherwise unexpectedly refreshing a new value every time.

The configuration may also help here so it’s clear what the relationship is between these resources. There is at least one change triggering things here, so you need to pinpoint what that is. Is there a configuration field which does not match the actual state? If a resource is causing a perpetual diff, you can see the complete details from a saved planfile using terraform show -json planfile.

Hi @jbardin ,

sorry about the hidden . Here is the original for that part:

  ~ resource "aws_eks_cluster" "this" {
        id                            = "projectname"
        name                          = "projectname"
        tags                          = {
            "terraform-aws-modules" = "eks"
        }
        # (14 unchanged attributes hidden)

      ~ encryption_config {
            # (1 unchanged attribute hidden)

          ~ provider {
              ~ key_arn = "arn:aws:kms:eu-west-1:***:key/8bb77903-2b18-4fdb-8a02-58bfdf59c90a" -> "arn:aws:kms:eu-west-1:***:key/401edf84-ee75-4920-abdb-46f5ede936fd"
            }
        }

        # (5 unchanged blocks hidden)
    }

the value is different as you can seen, but if I open th aws console the value that is there is:

KMS key ID
8bb77903-2b18-4fdb-8a02-58bfdf59c90a 

The other key was created by the “eks” module when it was updated after that

kms_key_enable_default_policy = false

was put so it should use the:

encryption_config = {
    provider_key_arn = aws_kms_key.kubernetes.arn
    resources        = ["secrets"]
  }

PS: The correct current value is 8bb77903-2b18-4fdb-8a02-58bfdf59c90a for the key

But the apply makes no effect.

Edit:

And this is what it shows it the show -json related with that change

{
    "address": "module.custom_eks.module.eks.aws_eks_cluster.this[0]",
    "mode": "managed",
    "type": "aws_eks_cluster",
    "name": "this",
    "index": 0,
    "provider_name": "registry.terraform.io/hashicorp/aws",
    "schema_version": 1,
        "encryption_config": [
            {
                "provider": [
                    {
                        "key_arn": "arn:aws:kms:eu-west-1:***:key/401edf84-ee75-4920-abdb-46f5ede936fd"
                    }
                ],
                "resources": [
                    "secrets"
                ]
            }
        ],

Edit 2:

-refresh=false makes the exact same result

terraform plan -refresh=false



Terraform will perform the following actions:

  # module.custom_eks.module.eks.data.aws_eks_addon_version.this["coredns"] will be read during apply
  # (depends on a resource or a module with changes pending)
 <= data "aws_eks_addon_version" "this" {
      + addon_name         = "coredns"
      + id                 = (known after apply)
      + kubernetes_version = "1.32"
      + most_recent        = true
      + region             = (known after apply)
      + version            = (known after apply)
    }

  # module.custom_eks.module.eks.data.aws_eks_addon_version.this["eks-pod-identity-agent"] will be read during apply
  # (depends on a resource or a module with changes pending)
 <= data "aws_eks_addon_version" "this" {
      + addon_name         = "eks-pod-identity-agent"
      + id                 = (known after apply)
      + kubernetes_version = "1.32"
      + most_recent        = true
      + region             = (known after apply)
      + version            = (known after apply)
    }

  # module.custom_eks.module.eks.data.aws_eks_addon_version.this["kube-proxy"] will be read during apply
  # (depends on a resource or a module with changes pending)
 <= data "aws_eks_addon_version" "this" {
      + addon_name         = "kube-proxy"
      + id                 = (known after apply)
      + kubernetes_version = "1.32"
      + most_recent        = true
      + region             = (known after apply)
      + version            = (known after apply)
    }

  # module.custom_eks.module.eks.data.tls_certificate.this[0] will be read during apply
  # (depends on a resource or a module with changes pending)
 <= data "tls_certificate" "this" {
      + certificates = (known after apply)
      + id           = (known after apply)
      + url          = "https://oidc.eks.eu-west-1.amazonaws.com/id/[[HIDDEN]]"
    }

  # module.custom_eks.module.eks.aws_eks_addon.this["coredns"] will be updated in-place
  ~ resource "aws_eks_addon" "this" {
      ~ addon_version               = "v1.11.4-eksbuild.24" -> (known after apply)
        id                          = "[[HIDDEN]]:coredns"
        tags                        = {}
        # (12 unchanged attributes hidden)

        # (1 unchanged block hidden)
    }

  # module.custom_eks.module.eks.aws_eks_addon.this["eks-pod-identity-agent"] will be updated in-place
  ~ resource "aws_eks_addon" "this" {
      ~ addon_version               = "v1.3.10-eksbuild.1" -> (known after apply)
        id                          = "[[HIDDEN]]:eks-pod-identity-agent"
        tags                        = {}
        # (12 unchanged attributes hidden)

        # (1 unchanged block hidden)
    }

  # module.custom_eks.module.eks.aws_eks_addon.this["kube-proxy"] will be updated in-place
  ~ resource "aws_eks_addon" "this" {
      ~ addon_version               = "v1.32.9-eksbuild.2" -> (known after apply)
        id                          = "[[HIDDEN]]:kube-proxy"
        tags                        = {}
        # (12 unchanged attributes hidden)

        # (1 unchanged block hidden)
    }

  # module.custom_eks.module.eks.aws_eks_cluster.this[0] will be updated in-place
  ~ resource "aws_eks_cluster" "this" {
        id                            = "[[HIDDEN]]"
        name                          = "[[HIDDEN]]"
        tags                          = {
            "terraform-aws-modules" = "eks"
        }
        # (14 unchanged attributes hidden)

      ~ encryption_config {
            # (1 unchanged attribute hidden)

          ~ provider {
              ~ key_arn = "arn:aws:kms:eu-west-1:[[HIDDEN]]:key/8bb77903-2b18-4fdb-8a02-58bfdf59c90a" -> "arn:aws:kms:eu-west-1:[[HIDDEN]]:key/401edf84-ee75-4920-abdb-46f5ede936fd"
            }
        }

        # (5 unchanged blocks hidden)
    }

  # module.custom_eks.module.eks.aws_iam_openid_connect_provider.oidc_provider[0] will be updated in-place
  ~ resource "aws_iam_openid_connect_provider" "oidc_provider" {
        id              = "arn:aws:iam::[[HIDDEN]]:oidc-provider/oidc.eks.eu-west-1.amazonaws.com/id/[[HIDDEN]]"
        tags            = {
            "Name" = "[[HIDDEN]]-eks-irsa"
        }
      ~ thumbprint_list = [
          - "",
        ] -> (known after apply)
        # (4 unchanged attributes hidden)
    }

Plan: 0 to add, 5 to change, 0 to destroy.
╷

I’m still unsure what the overall dependencies are here without the config, but I’m going to go with the theory that key_arn is really what is causing the issue here, and the rest is just additional noise.

I think if -refresh=false still shows the change, then you should see warnings in the logs about that resource not applying the correct value. It seems this was already reported in the provider repo, but was closed since there was no follow up. If this is still the case with a new release, you might want to file a new issue with a minimal reproduction.

1 Like

Thank you so much! You were absolutely right. The issue was exactly as you pointed out. After reviewing the documentation for the “eks” module, I found an input called create_kms_key. By setting this to false, the additional key that was being created was destroyed, and Terraform stopped trying to switch the KMS key. That resolved my problem. I really appreciate your help!