Objects have changed outside of Terraform

Terraform
1

Good evening,

I’m using the below data source to obtain a list of users to be added to a principals field in “aws_iam_policy_document”. Unfortunately every time I execute a plan/apply TF is reporting changes were made outside of Terraform due to the principals being reordered.

Is there anyway to code around this, or is this simply b/c AWS randomly orders principals regardless of the input order? For the record I tried using “sort” on the data source result in a local variable and then passed that to the policy document. No dice.

data "aws_iam_group" "readonly" {
  group_name = "readonly"
}

data "aws_iam_policy_document" 
...
    principals {
      type        = "AWS"
      identifiers = data.aws_iam_group.readonly.users[*].arn
    }
...

> terraform plan
Terraform detected the following changes made outside of Terraform since the last "terraform apply":
...
~ {
                      ~ Principal = {
                          ~ AWS = [
                              + "arn:aws-us-gov:iam::999999999:user/ghi"
                              + "arn:aws-us-gov:iam::999999999:user/def",
                              + "arn:aws-us-gov:iam::999999999:user/abc",
                              - "arn:aws-us-gov:iam::999999999:user/ghi"
...

Thank you for your assistance.

2

Hi @Kimmel,

Unfortunately indeed AWS APIs do tend to normalize IAM policies in various ways when returning a document that you previously submitted in another request. The AWS provider is sometimes able to correctly classify those changes as normalization and not significant, but it’s not always possible and so that can lead to situations like this where the remote object is different in a way that we know as humans is equivalent, but Terraform doesn’t know that.

The usual way to avoid this sort of notification is to order the input to match the normalized form the remote API expects. It sounds like you already tried to use sort here and it didn’t help, which suggests that the API is returning it in some order other than what the sort function creates (a lexical sort by unicode code unit).

Can you share the full diff Terraform returned for that particular object? I’d like to see specifically which resource type this is and what order the remote API returned vs. what order these appear in your input.

3 
Note: Objects have changed outside of Terraform

Terraform detected the following changes made outside of Terraform since the last "terraform apply":

  # aws_kms_key.cloudwatch_notifications has been changed
  ~ resource "aws_kms_key" "cloudwatch_notifications" {
        id                                 = "###########################"
      ~ policy                             = jsonencode(
          ~ {
              ~ Statement = [
                    {
                        Action    = "kms:*"
                        Effect    = "Allow"
                        Principal = {
                            AWS = "arn:aws:iam::999999999999:root"
                        }
                        Resource  = "*"
                        Sid       = "root_access"
                    },
                  ~ {
                      ~ Principal = {
                          ~ AWS = [
                              - "arn:aws:iam::999999999999:user/abc",
                              - "arn:aws:iam::999999999999:user/def",
                              - "arn:aws:iam::999999999999:user/ghi",
                              - "arn:aws:iam::999999999999:user/readonly",
                              - "arn:aws:iam::999999999999:user/jkl",
                              - "arn:aws:iam::999999999999:user/mno",
                              - "arn:aws:iam::999999999999:user/pqr",
                              + "arn:aws:iam::999999999999:user/stu",
                              + "arn:aws:iam::999999999999:user/vwx",
                                "arn:aws:iam::999999999999:user/yz",
                              - "arn:aws:iam::999999999999:user/abc1",
                              - "arn:aws:iam::999999999999:user/def1",
                              + "arn:aws:iam::999999999999:user/jkl1",
                                "arn:aws:iam::999999999999:user/mno1",
                                "arn:aws:iam::999999999999:user/pqr1",
                              - "arn:aws:iam::999999999999:user/vwx",
                              + "arn:aws:iam::999999999999:user/readonly",
                                "arn:aws:iam::999999999999:user/ghi1",
                              + "arn:aws:iam::999999999999:user/def1",
                              + "arn:aws:iam::999999999999:user/def",
                              + "arn:aws:iam::999999999999:user/abc1",
                              + "arn:aws:iam::999999999999:user/stu1",
                              + "arn:aws:iam::999999999999:user/abc",
                              + "arn:aws:iam::999999999999:user/ghi",
                              + "arn:aws:iam::999999999999:user/jkl",
                                "arn:aws:iam::999999999999:user/vwx1",
                                "arn:aws:iam::999999999999:user/yz1",
                              - "arn:aws:iam::999999999999:user/stu",
                              - "arn:aws:iam::999999999999:user/jkl1",
                              - "arn:aws:iam::999999999999:user/stu1",
                              + "arn:aws:iam::999999999999:user/pqr",
                              + "arn:aws:iam::999999999999:user/mno"
                            ]
                        }
                        # (4 unchanged elements hidden)
                    },
                    {
                        Action    = [
                            "kms:GenerateDataKey*",
                            "kms:Decrypt",
                        ]
                        Effect    = "Allow"
                        Principal = {
                            Service = "cloudwatch.amazonaws.com"
                        }
                        Resource  = "*"
                        Sid       = "cloudwatch_encrypt_access"
                    },
                ]
                # (1 unchanged element hidden)
            }
        )
        tags                               = {
            "Environment" = "dev"
            "Role"        = "cloudwatch"
            "Terraform"   = "true"
        }
        # (9 unchanged attributes hidden)
    }

Unless you have made equivalent changes to your configuration, or ignored the relevant attributes using ignore_changes, the following plan may include actions to undo or respond to these changes.

──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

No changes. Your infrastructure matches the configuration.

Your configuration already matches the changes detected above. If you'd like to update the Terraform state to match, create and apply a refresh-only plan:
  terraform apply -refresh-only

The obfuscation of the names above is consistent - i.e. “smithj” is “abc” for both “+” and “-”.

The above plan is after a new installation from a sorted local being passed to the principal.

locals {
  sorted_readonly = sort(data.aws_iam_group.readonly.users[*].arn)
}

data "aws_iam_group" "readonly" {
  group_name = "readonly"
}
4

Hi @Kimmel,

I appreciate that you need to obfuscate this in order to avoid revealing identifying information about people who have accounts in your AWS account, though unfortunately it does mean that I can’t use this result to understand what sort of ordering, if any, the remote API is imposing on those ARNs. :confused:

Looking at the real output that you can see yourself, if you exclude the ones that were removed - from consideration, and look at only the ones that were either unchanged or added +, does it look like the ARNs are in any particular systematic order, or does it just seem to be arbitrary?

My reason for asking is that I’m trying to understand whether sort is even a viable option here, which would require the API to be returning them in a lexical-sorted order. If they’re being returned in some other order then I don’t think there’ll be any good way to match it via a Terraform expression and instead it’d only be fixable by adding the missing normalization-detection rule to the AWS provider so it can treat these statements as equivalent regardless of ARN ordering. :confused:

5

Good evening @apparentlymart,

I was afraid the obfuscation would limit your diagnostic options. I cannot determine any systematic ordering. Below are a handful of the first adds and unchanged.

+  /rsAAA
+  /bhAAAA
   /coAAAA
+  /klAAAAA
   /wlAAAA
   /moAAAAA
+  /AAAAAAAAAAA
   /chAAA

Do you see anything in the above?