Terraform will update resource even if the values didn't changed

aadipop · September 6, 2023, 6:50am

Hello,

I have a config that get secrets from AWS SecretManager and creates k8s secrets and from time to time, especially when I modify some values in terraform-aws-modules/eks/aws all secrets will be recreated even if the values are not changed

This is the resource definition:

resource "kubernetes_secret" "app-configs" {
  for_each = {
    for app in var.apps_secrets : app.name => {
      name        = app.name
      namespace   = app.ns
      secrets     = {
        for secret in app.secrets: upper(secret) => jsondecode(data.aws_secretsmanager_secret_version.app_configs_current.secret_string)["${app.ns}-${secret}"]
      }
    }
  }

  metadata {
    name                = each.value.name
    namespace           = "${each.value.namespace}"
  }

  data = each.value.secrets
}

and this is the tf output:

  # module.helm.kubernetes_secret.app-configs["my-app"] will be updated in-place
  ~ resource "kubernetes_secret" "app-configs" {
      ~ data                           = (sensitive value)
        id                             = "app/my-app"
        # (3 unchanged attributes hidden)

        # (1 unchanged block hidden)
    }

any hints?

thanks,
Adrian

aadipop · September 6, 2023, 6:56am

Also, I have this issue too but I can’t change the templatefile() function to file() because I use variables in those templates

the resource definition for this one is:

resource "aws_iam_policy" "this" {
  policy = templatefile("./modules/irsa/iam_policies/${var.policy_name}.json", var.configs)
  name   = var.policy_name
}

and an example policy is:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "ec2:CreateKeyPair",
                "iam:CreateRole",
                "logs:CreateLogGroup"
            ],
            "Resource": "*"
        }
    ]
}

also this recreate is triggered by value changes in terraform-aws-modules/eks/aws

apparentlymart · September 6, 2023, 8:11pm

Hi @aadipop,

According to this plan output, “data” has changed, but Terraform isn’t showing how it has changed because the provider has marked this argument as being “sensitive”.

One way to debug this would be to ask Terraform to generate a machine-readable (JSON) plan description, since machine-readable output does not hide sensitive values:

terraform plan -out=tfplan
terraform show -json tfplan

If you pretty-print the JSON output and load it into a text editor you can hopefully find the JSON equivalent of the change description you shared in your original question. It should include both the old and new values of data so that you can compare the two and see how they differ.

If you’re not sure how to interpret the JSON plan description then please share what you find (making sure to remove any parts that actually are private) and then I or someone else can hopefully help to interpret it.

aadipop · October 12, 2023, 6:12am

Hello,

Thanks, I found the differences but idk how to solve them …

For first one related to kubernetes_secret, when secrets are marked to update, the difference is the metadata field:

"before": {
      ... same as "after"
    },
    "after": {
      ... same as "before"
    },
    "after_unknown": {},
    "before_sensitive": {
      "binary_data": true,
      "data": true,
      "metadata": [{}]                                                       <<<--------------------
    },
    "after_sensitive": {
      "binary_data": true,
      "data": true,
      "metadata": [{ "annotations": {}, "labels": {} }]   <<<--------------------
    }

and for IAM Roles the assume_role_policy field is present in before but not in after:

"before": {
      "arn": "arn:aws:iam::000000000000:role/AwsEksAutoscaler",
      "assume_role_policy": "{\"Statement\":[{\"Action\":\"sts:AssumeRoleWithWebIdentity\",\"Condition\":{\"StringEquals\":{\"oidc.eks.eu-west-1.amazonaws.com/id/000000000000:aud\":\"sts.amazonaws.com\"},\"StringLike\":{\"oidc.eks.eu-west-1.amazonaws.com/id/000000000000:sub\":\"system:serviceaccount:kube-system:cluster-autoscaler\"}},\"Effect\":\"Allow\",\"Principal\":{\"Federated\":\"arn:aws:iam::000000000000:oidc-provider/oidc.eks.eu-west-1.amazonaws.com/id/000000000000\"},\"Sid\":\"\"}],\"Version\":\"2012-10-17\"}",
      "create_date": "2023-08-07T06:44:51Z",
      ...
    },
    "after": {
      "arn": "arn:aws:iam::000000000000:role/AwsEksAutoscaler",
      "create_date": "2023-08-07T06:44:51Z",
      ...
    },

those resources are marked for update in place only if I make some changes to terraform-aws-modules/eks/aws or terraform-aws-modules/vpc/aws modules

Thanks,
Adrian

aadipop · October 12, 2023, 7:38am

the assume_role_policy comes from here:

data "aws_iam_policy_document" "this" {
  statement {
    actions = ["sts:AssumeRoleWithWebIdentity"]
    effect  = "Allow"

    principals {
      identifiers = [var.oidc_provider_arn]
      type        = "Federated"
    }

    condition {
      test     = "StringLike"
      variable = "${replace(var.oidc_provider_url, "https://", "")}:sub"
      values   = var.resource
    }

    condition {
      test     = "StringEquals"
      variable = "${replace(var.oidc_provider_url, "https://", "")}:aud"
      values   = ["sts.amazonaws.com"]
    }
  }
}

resource "aws_iam_role" "this" {
  assume_role_policy = data.aws_iam_policy_document.this.json
  name               = var.role_name
  description        = var.role_description
}

aadipop · October 12, 2023, 7:44am

and from tfplan looks like the before data is seen as null

{
  "address": "module.helm.module.asg_controller_irsa.data.aws_iam_policy_document.this",
  "module_address": "module.helm.module.asg_controller_irsa",
  "mode": "data",
  "type": "aws_iam_policy_document",
  "name": "this",
  "provider_name": "registry.terraform.io/hashicorp/aws",
  "change": {
    "actions": ["read"],
    "before": null,
    "after": {
      "override_json": null,
      "override_policy_documents": null,
      "policy_id": null,
      "source_json": null,
      "source_policy_documents": null,
      "statement": [
        {
          "actions": ["sts:AssumeRoleWithWebIdentity"],
          "condition": [
            { "test": "StringEquals", "values": ["sts.amazonaws.com"] },
            {
              "test": "StringLike",
              "values": [
                "system:serviceaccount:kube-system:cluster-autoscaler"
              ]
            }
          ],
          "effect": "Allow",
          "not_actions": null,
          "not_principals": [],
          "not_resources": null,
          "principals": [
            {
              "identifiers": [
                "arn:aws:iam::000000000000:oidc-provider/oidc.eks.eu-west-1.amazonaws.com/id/000000000000"
              ],
              "type": "Federated"
            }
          ],
          "resources": null,
          "sid": null
        }
      ],
      "version": null
    },
    "after_unknown": {
      "id": true,
      "json": true,
      "statement": [
        {
          "actions": [false],
          "condition": [
            { "values": [false], "variable": true },
            { "values": [false], "variable": true }
          ],
          "not_principals": [],
          "principals": [{ "identifiers": [false] }]
        }
      ]
    },
    "before_sensitive": false,
    "after_sensitive": {
      "statement": [
        {
          "actions": [false],
          "condition": [{ "values": [false] }, { "values": [false] }],
          "not_principals": [],
          "principals": [{ "identifiers": [false] }]
        }
      ]
    }
  },
  "action_reason": "read_because_config_unknown"
},

apparentlymart · October 12, 2023, 3:38pm

Hi @aadipop,

Thanks for sharing the JSON plan snippets.

Unfortunately I was expecting to see a change between “before” and “after” in the kubernetes_secret resource, because that was shown has having its data change compared to the prior value.

The changes between before_sensitive and after_sensitive are curious but I think ultimately not important, since only the leaf boolean values are meaningful in those data structures: the presence of a true indicates that the corresponding values in before or after should be treated as “sensitive”, but empty objects there would have no effect because nothing is marked as true.

With that said then, I’m honestly unsure what to suggest next. It seems like there is a difference here that even the JSON plan representation can’t understand. One way that could happen is if the types of some values have changed in a way that doesn’t affect the JSON serialization, such as if an object value were replaced with a map value that has the same key/value pairs, or vice-versa: Terraform distinguishes object types from map types, but JSON cannot. It’s odd though, because in modern Terraform the human-oriented plan output is built from the JSON version and so I wouldn’t expect the UI to show a change that the JSON serialization does not.

Just to confirm that nothing has changed in the meantime, can you try terraform show tfplan (no -json this time) on the same plan file and confirm that it still shows the proposed change to the kubernetes_secret resource? I’m wondering if something changed in the meantime that has changed the planning outcome.

aadipop · November 2, 2023, 6:18am

Hello again

sorry for late response,

I saw this behaviour also when I apply changes from github actions ( ubuntu vm ) and after apply from my pc ( osx ), or vice-versa

maybe the ubuntu os will fetch data different from aws secrets manager and templatefile()

*we start applying the plan only from the pipeline and is pretty stable with clean outputs

but i still think that the terraform-aws-modules/eks/aws have something to do with that … because any change on this, even the smallest tag insert will trigger the replace for all secrets and policies

I’ll try to make some tests till next week

thanks,
Adrian

aadipop · November 2, 2023, 6:35am

Plan: 0 to add, 124 to change, 0 to destroy. and then the output: Apply complete! Resources: 0 added, 1 changed, 0 destroyed.

I apply the plan from GitHub Action’s vm then run terraform apply from my pc without any changes

I think I’ll start trying with migrate the eks-module to basic resources

punia_suresh · March 26, 2024, 6:34am

I have also similar issue with eks cluster, eks cluster itself going to recreate. plan showing 4 addition 23 changes and 4 destroy that include eks cluster deletion, below the plan output in pretty-json, please help in diagnosing this issue

{
“format_version”: “0.1”,
“terraform_version”: “0.13.6”,
“variables”: {
“account_number”: {
“value”: “xxxxxxxxxxxxx”
},
“account_team_name”: {
“value”: “atlas”
},
“aws_region”: {
“value”: “ap-southeast-2”
},
“resource_changes”: [
{
“address”: “module.eks.aws_eks_cluster.this[0]”,
“module_address”: “module.eks”,
“mode”: “managed”,
“type”: “aws_eks_cluster”,
“name”: “this”,
“index”: 0,
“provider_name”: “Terraform Registry”,
“change”: {
“actions”: [
“create”,
“delete”
],
“before”: {
“arn”: “arn:aws:eks:ap-southeast-2:xxxxxxxxxxxxx:cluster/vz-vss-atlas-int1-ap-southeast-2-eks-v3-new”,
“certificate_authority”: [
{
“data”: “LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM1ekNDQWMrZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJeU1EZ3dNakl3TWpZME4xb1hEVE15TURjek1ESXdNalkwTjFvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTDFDCk1DTlVpMEk3OGtZK1g1SHQ5NzhNMUZrbThpYy9ZU3U2QmJWeVNUditUcUMzSlVTbkM3V0JMTVlSbHY1aDFKT3MKTVV0bEhlOFQ2T2NPNy9FK1V2Mm14SmxtNGZTRExNMHRxbSs3MEwwUFQzUkZpd2ZxOWVTTVdUeW50b2FORUQzMgpuZjRJdjhCdjdZUCtkdEFrRXF5V0xYb08xUi9ZUERTeWQ0R09IZkhYdkVGbzhUSzdsclIrYjhSTHlJWTZxV05GClBvTDZLbktoN1RQZUpuOFZURHlobVhnTTZ1QlUwcUx5cGFTSDIySTY5ZUYzWThoTFNoa0FobWpZOWtJNnBlTlUKLys0YnlGSkhUMzhoMHVId3dSaTdkaXdYb0tmbDBqeHRLN0V6OERuUUxrcGdtVGs3NnhMRGZkcDNCL1ljOXNFbgpKZXR5eFJLbTVWbExnZ2owRUc4Q0F3RUFBYU5DTUVBd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0hRWURWUjBPQkJZRUZNdnJaSG1oWS8vSWxEOXN3QjlMZmhNZXd3MTBNQTBHQ1NxR1NJYjMKRFFFQkN3VUFBNElCQVFDVmVUaTFuQittTUFDTjlCWi9iTVZ2cUJweUFzd2lmbjV3c2dZY0hSR2pRS04rOXB4UAozR3hTc3dySGlDUDFoUWpVNTlxNmI1YmhiQzV6dG12blE3VnpYOGpDSWtDc1lnNUY0WDl4aDFWZlRoLzNUK2hUCmt4RDBiWnNEV3A2VHVrK0UyWlBkQ084TGgxQlNqOTZXQ25ZQlRPcVBZQkhJWjVRNFN4SEhENkpnUTlMelVDUHYKd2VwSS9hY3hjK0t1eUNkcUE3VUUvamFUOGlXaWVYOEJVTVcreGx2VmpWVStDa0lYTmxPaVJvRStiOXN0WU1jSAp4L25uZkxBN1NBYnZveWZ1K0VocStLbFJuMUpzY0Q2ODVNZERMZXI0OGdvbSt1bm81bGpPRkFVZWpvYmFhSFNuCitXVWNBSCtoaWk0UWVwaHRNTmw4dTluVGRDeW8vL1U1WW5EOQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==”
}
],
“created_at”: “2022-08-02 20:22:10.018 +0000 UTC”,
“enabled_cluster_log_types”: [
“api”,
“audit”,
“authenticator”,
“controllerManager”,
“scheduler”
],
“encryption_config”: [
{
“provider”: [
{
“key_arn”: “arn:aws:kms:ap-southeast-2:xxxxxxxxxxxxx:key/6a5a416a-9a37-446b-98e7-864bfe399ced”
}
],
“resources”: [
“secrets”
]
}
],
“endpoint”: “https://E85C22AD359CEED323AC015FAEAF0EE7.gr7.ap-southeast-2.eks.amazonaws.com”,
“id”: “vz-vss-atlas-int1-ap-southeast-2-eks-v3-new”,
“identity”: [
{
“oidc”: [
{
“issuer”: “https://oidc.eks.ap-southeast-2.amazonaws.com/id/E85C22AD359CEED323AC015FAEAF0EE7”
}
]
}
],
“kubernetes_network_config”: [
{
“ip_family”: “ipv4”,
“service_ipv4_cidr”: “172.20.0.0/16”
}
],
“name”: “vz-vss-atlas-int1-ap-southeast-2-eks-v3-new”,
“platform_version”: “eks.9”,
“role_arn”: “arn:aws:iam::xxxxxxxxxxxxx:role/vz-vss-atlas-int1-eks-v3-new-ap-southeast-2-cluster-role”,
“status”: “ACTIVE”,
“tags”: {
“App”: “Atlas”,
“Createdby”: “Terraform”,
“Environment”: “Nonprod”,
“Level”: “Int”,
“Name”: “vz-vss-atlas-int1-ap-southeast-2-eks-v3-new”,
“Organization”: “VSS”,
“Owner”: “vz-vss-aws-di@xyz.com”,
“Purpose”: “EKS cluster”,
“Role”: “App”,
“Team”: “Delivery Infrastructure”,
“Userid”: “v918556”,
“VSAD”: “HLKV”
},
“tags_all”: {
“App”: “Atlas”,
“Createdby”: “Terraform”,
“Environment”: “Nonprod”,
“Level”: “Int”,
“Name”: “vz-vss-atlas-int1-ap-southeast-2-eks-v3-new”,
“Organization”: “VSS”,
“Owner”: “vz-vss-aws-di@xyz.com”,
“Purpose”: “EKS cluster”,
“Role”: “App”,
“Team”: “Delivery Infrastructure”,
“Userid”: “v918556”,
“VSAD”: “HLKV”
},
“timeouts”: {
“create”: “30m”,
“delete”: “15m”,
“update”: null
},
“version”: “1.28”,
“vpc_config”: [
{
“cluster_security_group_id”: “sg-0f776df865ed99267”,
“endpoint_private_access”: true,
“endpoint_public_access”: false,
“public_access_cidrs”: ,
“security_group_ids”: [
“sg-094218aa3b0c63f06”
],
“subnet_ids”: [
“subnet-0660b49c2db3b4e95”,
“subnet-0666c40c49e865b7a”,
“subnet-0f3a5e62d1d3052c2”
],
“vpc_id”: “vpc-06ee8f772c5016a7c”
}
]
},
“after”: {
“enabled_cluster_log_types”: [
“api”,
“audit”,
“authenticator”,
“controllerManager”,
“scheduler”
],
“encryption_config”: [
{
“provider”: [
{
“key_arn”: “arn:aws:kms:ap-southeast-2:xxxxxxxxxxxxx:key/6a5a416a-9a37-446b-98e7-864bfe399ced”
}
],
“resources”: [
“secrets”
]
}
],
“name”: “vz-vss-atlas-int1-ap-southeast-2-eks-v3-new”,
“tags”: {
“App”: “Atlas”,
“Createdby”: “Terraform”,
“Environment”: “Nonprod”,
“Level”: “Int”,
“Name”: “vz-vss-atlas-int1-ap-southeast-2-eks-v3-new”,
“Organization”: “VSS”,
“Owner”: “vz-vss-aws-di@xyz.com”,
“Purpose”: “EKS cluster”,
“Role”: “App”,
“Team”: “Delivery Infrastructure”,
“Userid”: “v918556”,
“Vsad”: “HLKV”
},
“tags_all”: {
“App”: “Atlas”,
“Createdby”: “Terraform”,
“Environment”: “Nonprod”,
“Level”: “Int”,
“Name”: “vz-vss-atlas-int1-ap-southeast-2-eks-v3-new”,
“Organization”: “VSS”,
“Owner”: “vz-vss-aws-di@xyz.com”,
“Purpose”: “EKS cluster”,
“Role”: “App”,
“Team”: “Delivery Infrastructure”,
“Userid”: “v918556”,
“Vsad”: “HLKV”
},
“timeouts”: {
“create”: “30m”,
“delete”: “15m”,
“update”: null
},
“version”: “1.28”,
“vpc_config”: [
{
“endpoint_private_access”: true,
“endpoint_public_access”: false,
“security_group_ids”: [
“sg-094218aa3b0c63f06”
],
“subnet_ids”: [
“subnet-0660b49c2db3b4e95”,
“subnet-0666c40c49e865b7a”,
“subnet-0f3a5e62d1d3052c2”
]
}
]
},
“after_unknown”: {
“arn”: true,
“certificate_authority”: true,
“created_at”: true,
“enabled_cluster_log_types”: [
false,
false,
false,
false,
false
],
“encryption_config”: [
{
“provider”: [
{}
],
“resources”: [
false
]
}
],
“endpoint”: true,
“id”: true,
“identity”: true,
“kubernetes_network_config”: true,
“platform_version”: true,
“role_arn”: true,
“status”: true,
“tags”: {},
“tags_all”: {},
“timeouts”: {},
“vpc_config”: [
{
“cluster_security_group_id”: true,
“public_access_cidrs”: true,
“security_group_ids”: [
false
],
“subnet_ids”: [
false,
false,
false
],
“vpc_id”: true
}
]
}
}
},
{
“address”: “module.eks.aws_iam_openid_connect_provider.oidc_provider[0]”,
“module_address”: “module.eks”,
“mode”: “managed”,
“type”: “aws_iam_openid_connect_provider”,
“name”: “oidc_provider”,
“index”: 0,
“provider_name”: “Terraform Registry”,
“change”: {
“actions”: [
“delete”,
“create”
],
“before”: {
“arn”: “arn:aws:iam::xxxxxxxxxxxxx:oidc-provider/oidc.eks.ap-southeast-2.amazonaws.com/id/E85C22AD359CEED323AC015FAEAF0EE7”,
“client_id_list”: [
“sts.amazonaws.com”
],
“id”: “arn:aws:iam::xxxxxxxxxxxxx:oidc-provider/oidc.eks.ap-southeast-2.amazonaws.com/id/E85C22AD359CEED323AC015FAEAF0EE7”,
“tags”: {
“Environment”: “Nonprod”,
“Name”: “arn:aws:iam::xxxxxxxxxxxxx:oidc-provider/oidc.eks.ap-southeast-2.amazonaws.com/id/E85C22AD359CEED323AC015FAEAF0EE7”,
“Owner”: “vz-vss-aws-di@one.xyz.com”,
“Team”: “DI”,
“Userid”: “v553144”,
“Vsad”: “HLKV”
},
“tags_all”: {
“Environment”: “Nonprod”,
“Name”: “arn:aws:iam::xxxxxxxxxxxxx:oidc-provider/oidc.eks.ap-southeast-2.amazonaws.com/id/E85C22AD359CEED323AC015FAEAF0EE7”,
“Owner”: “vz-vss-aws-di@one.xyz.com”,
“Team”: “DI”,
“Userid”: “v553144”,
“Vsad”: “HLKV”
},
“thumbprint_list”: [
“9e99a48a9960b14926bb7f3b02e22da2b0ab7280”
],
“url”: “oidc.eks.ap-southeast-2.amazonaws.com/id/E85C22AD359CEED323AC015FAEAF0EE7”
},
“after”: {
“tags”: null,
“thumbprint_list”: [
“9e99a48a9960b14926bb7f3b02e22da2b0ab7280”
]
},
“after_unknown”: {
“arn”: true,
“client_id_list”: true,
“id”: true,
“tags_all”: true,
“thumbprint_list”: [
false
],
“url”: true
}
}
},

aadipop · March 29, 2024, 9:22pm

hey, a bit late response but … somehow I found the trigger, it’s related to eks module addons

cluster_addons = {
    coredns = {
      addon_version = "v1.10.1-eksbuild.7"
      # most_recent = true   <<< --- here
    }
    ...

If the latest addon version will change then will trigger a lot of changes
idk exactly why but with hardcoded addon_version works as expected

Thanks for helping Mart

Have a good day

aadipop · March 29, 2024, 9:25pm

without any changes to the plan? just between two terraform apply?

apparentlymart · April 12, 2024, 9:26pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Lifecycle and the existing resources Terraform	11	621	October 12, 2024
Digging in to what is being changed? Terraform	1	16	January 26, 2025
Override diffing when dealing with encrypted data Terraform Providers	2	223	August 27, 2021
Confused about aws-secretsmanager-secret-version resource behaviour AWS	1	4756	March 29, 2023
K8s provider resource version always changing Terraform Providers k8s	0	532	August 9, 2021

Terraform will update resource even if the values didn't changed

I have also similar issue with eks cluster, eks cluster itself going to recreate. plan showing 4 addition 23 changes and 4 destroy that include eks cluster deletion, below the plan output in pretty-json, please help in diagnosing this issue

Related topics