AWS KMS provider not working

Hi there,

We are having issues in creating AWS KMS key.

Terraform Version

Terraform v0.13.7

Terraform Configuration Files

resource "aws_kms_key" "a" {
  count                   = 1
  description             = "KMS key 1"
  deletion_window_in_days = 10
  policy                  = <<POLICY
      {
          "Statement": [
            {
                  "Sid": "Enable Admin IAM Permissions",
                  "Effect": "Allow",
                  "Principal": {
                      "AWS": ["arn:aws:iam::359816056456:role/developer"]
                  },
                  "Action": [
                      "kms:Create*",
                      "kms:Describe*",
                      "kms:Enable*",
                      "kms:List*",
                      "kms:Put*",
                      "kms:Update*",
                      "kms:Revoke*",
                      "kms:Disable*",
                      "kms:Get*",
                      "kms:Delete*",
                      "kms:ScheduleKeyDeletion",
                      "kms:CancelKeyDeletion",
                      "kms:TagResource",
                      "kms:UntagResource"],
                  "Resource": "*"
              }   
          ]
      }
    POLICY
}

Debug Output

Initializing the backend...

Initializing provider plugins...
- Using previously-installed hashicorp/aws v3.54.0

wal-miali-210423:kms-debug miali$ terraform apply
provider.aws.region
  The region where AWS operations will take place. Examples
  are us-east-1, us-west-2, etc.

  Enter a value: us-east-1

aws_kms_key.a[0]: Refreshing state... [id=2df3fe4a-fbca-480c-b182-5fd08bb7a4bb]

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
-/+ destroy and then create replacement

Terraform will perform the following actions:

  # aws_kms_key.a[0] is tainted, so must be replaced
-/+ resource "aws_kms_key" "a" {
      ~ arn                                = "arn:aws:kms:us-east-1:359816056456:key/2df3fe4a-fbca-480c-b182-5fd08bb7a4bb" -> (known after apply)
        bypass_policy_lockout_safety_check = false
        customer_master_key_spec           = "SYMMETRIC_DEFAULT"
        deletion_window_in_days            = 10
        description                        = "KMS key 1"
        enable_key_rotation                = false
      ~ id                                 = "2df3fe4a-fbca-480c-b182-5fd08bb7a4bb" -> (known after apply)
        is_enabled                         = true
      ~ key_id                             = "2df3fe4a-fbca-480c-b182-5fd08bb7a4bb" -> (known after apply)
        key_usage                          = "ENCRYPT_DECRYPT"
      ~ policy                             = jsonencode(
          ~ {
              ~ Statement = [
                  ~ {
                        Action    = [
                            "kms:Create*",
                            "kms:Describe*",
                            "kms:Enable*",
                            "kms:List*",
                            "kms:Put*",
                            "kms:Update*",
                            "kms:Revoke*",
                            "kms:Disable*",
                            "kms:Get*",
                            "kms:Delete*",
                            "kms:ScheduleKeyDeletion",
                            "kms:CancelKeyDeletion",
                            "kms:TagResource",
                            "kms:UntagResource",
                        ]
                        Effect    = "Allow"
                      ~ Principal = {
                          ~ AWS = "arn:aws:iam::359816056456:role/developer" -> [
                              + "arn:aws:iam::359816056456:role/developer",
                            ]
                        }
                        Resource  = "*"
                        Sid       = "Enable Admin IAM Permissions"
                    },
                ]
              - Version   = "2008-10-17" -> null
            }
        )
      - tags                               = {} -> null
      ~ tags_all                           = {} -> (known after apply)
    }

Plan: 1 to add, 0 to change, 1 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

Crash Output

Error: error waiting for KMS Key (8ed15ac6-9367-49ba-a93c-79d1e8e93e72) policy propagation: timeout while waiting for state to become 'TRUE' (last state: 'FALSE', timeout: 2m0s)

  on main.tf line 1, in resource "aws_kms_key" "a":
   1: resource "aws_kms_key" "a" {

Expected Behavior

KMS key is created.

Actual Behavior

Error: error waiting for KMS Key (8ed15ac6-9367-49ba-a93c-79d1e8e93e72) policy propagation: timeout while waiting for state to become 'TRUE' (last state: 'FALSE', timeout: 2m0s)

  on main.tf line 1, in resource "aws_kms_key" "a":
   1: resource "aws_kms_key" "a" {

Steps to Reproduce

Please list the full steps required to reproduce the issue, for example:

  1. terraform init
Initializing the backend...

Initializing provider plugins...
- Using previously-installed hashicorp/aws v3.54.0

wal-miali-210423:kms-debug miali$ terraform apply
provider.aws.region
  The region where AWS operations will take place. Examples
  are us-east-1, us-west-2, etc.

  Enter a value: us-east-1

aws_kms_key.a[0]: Refreshing state... [id=2df3fe4a-fbca-480c-b182-5fd08bb7a4bb]

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
-/+ destroy and then create replacement

Terraform will perform the following actions:

  # aws_kms_key.a[0] is tainted, so must be replaced
-/+ resource "aws_kms_key" "a" {
      ~ arn                                = "arn:aws:kms:us-east-1:359816056456:key/2df3fe4a-fbca-480c-b182-5fd08bb7a4bb" -> (known after apply)
        bypass_policy_lockout_safety_check = false
        customer_master_key_spec           = "SYMMETRIC_DEFAULT"
        deletion_window_in_days            = 10
        description                        = "KMS key 1"
        enable_key_rotation                = false
      ~ id                                 = "2df3fe4a-fbca-480c-b182-5fd08bb7a4bb" -> (known after apply)
        is_enabled                         = true
      ~ key_id                             = "2df3fe4a-fbca-480c-b182-5fd08bb7a4bb" -> (known after apply)
        key_usage                          = "ENCRYPT_DECRYPT"
      ~ policy                             = jsonencode(
          ~ {
              ~ Statement = [
                  ~ {
                        Action    = [
                            "kms:Create*",
                            "kms:Describe*",
                            "kms:Enable*",
                            "kms:List*",
                            "kms:Put*",
                            "kms:Update*",
                            "kms:Revoke*",
                            "kms:Disable*",
                            "kms:Get*",
                            "kms:Delete*",
                            "kms:ScheduleKeyDeletion",
                            "kms:CancelKeyDeletion",
                            "kms:TagResource",
                            "kms:UntagResource",
                        ]
                        Effect    = "Allow"
                      ~ Principal = {
                          ~ AWS = "arn:aws:iam::359816056456:role/developer" -> [
                              + "arn:aws:iam::359816056456:role/developer",
                            ]
                        }
                        Resource  = "*"
                        Sid       = "Enable Admin IAM Permissions"
                    },
                ]
              - Version   = "2008-10-17" -> null
            }
        )
      - tags                               = {} -> null
      ~ tags_all                           = {} -> (known after apply)
    }

Plan: 1 to add, 0 to change, 1 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes
2. `terraform apply`


### Additional Context
wal-miali-210423:kms-debug miali$ terraform init

Initializing the backend...

Initializing provider plugins...
- Using previously-installed hashicorp/aws v3.54.0

wal-miali-210423:kms-debug miali$ terraform apply
provider.aws.region
  The region where AWS operations will take place. Examples
  are us-east-1, us-west-2, etc.

  Enter a value: us-east-1

aws_kms_key.a[0]: Refreshing state... [id=2df3fe4a-fbca-480c-b182-5fd08bb7a4bb]

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
-/+ destroy and then create replacement

Terraform will perform the following actions:

  # aws_kms_key.a[0] is tainted, so must be replaced
-/+ resource "aws_kms_key" "a" {
      ~ arn                                = "arn:aws:kms:us-east-1:359816056456:key/2df3fe4a-fbca-480c-b182-5fd08bb7a4bb" -> (known after apply)
        bypass_policy_lockout_safety_check = false
        customer_master_key_spec           = "SYMMETRIC_DEFAULT"
        deletion_window_in_days            = 10
        description                        = "KMS key 1"
        enable_key_rotation                = false
      ~ id                                 = "2df3fe4a-fbca-480c-b182-5fd08bb7a4bb" -> (known after apply)
        is_enabled                         = true
      ~ key_id                             = "2df3fe4a-fbca-480c-b182-5fd08bb7a4bb" -> (known after apply)
        key_usage                          = "ENCRYPT_DECRYPT"
      ~ policy                             = jsonencode(
          ~ {
              ~ Statement = [
                  ~ {
                        Action    = [
                            "kms:Create*",
                            "kms:Describe*",
                            "kms:Enable*",
                            "kms:List*",
                            "kms:Put*",
                            "kms:Update*",
                            "kms:Revoke*",
                            "kms:Disable*",
                            "kms:Get*",
                            "kms:Delete*",
                            "kms:ScheduleKeyDeletion",
                            "kms:CancelKeyDeletion",
                            "kms:TagResource",
                            "kms:UntagResource",
                        ]
                        Effect    = "Allow"
                      ~ Principal = {
                          ~ AWS = "arn:aws:iam::359816056456:role/developer" -> [
                              + "arn:aws:iam::359816056456:role/developer",
                            ]
                        }
                        Resource  = "*"
                        Sid       = "Enable Admin IAM Permissions"
                    },
                ]
              - Version   = "2008-10-17" -> null
            }
        )
      - tags                               = {} -> null
      ~ tags_all                           = {} -> (known after apply)
    }

Plan: 1 to add, 0 to change, 1 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

aws_kms_key.a[0]: Destroying... [id=2df3fe4a-fbca-480c-b182-5fd08bb7a4bb]
aws_kms_key.a[0]: Destruction complete after 0s
aws_kms_key.a[0]: Creating...
aws_kms_key.a[0]: Still creating... [10s elapsed]
aws_kms_key.a[0]: Still creating... [20s elapsed]
aws_kms_key.a[0]: Still creating... [30s elapsed]
aws_kms_key.a[0]: Still creating... [40s elapsed]
aws_kms_key.a[0]: Still creating... [50s elapsed]
aws_kms_key.a[0]: Still creating... [1m0s elapsed]
aws_kms_key.a[0]: Still creating... [1m10s elapsed]
aws_kms_key.a[0]: Still creating... [1m20s elapsed]
aws_kms_key.a[0]: Still creating... [1m30s elapsed]
aws_kms_key.a[0]: Still creating... [1m40s elapsed]
aws_kms_key.a[0]: Still creating... [1m50s elapsed]
aws_kms_key.a[0]: Still creating... [2m0s elapsed]

Error: error waiting for KMS Key (8ed15ac6-9367-49ba-a93c-79d1e8e93e72) policy propagation: timeout while waiting for state to become 'TRUE' (last state: 'FALSE', timeout: 2m0s)

  on main.tf line 1, in resource "aws_kms_key" "a":
   1: resource "aws_kms_key" "a" {

I also have been having this happen recently. The strange thing is, is the key is created with a valid policy according to the AWS console.

Edit: Seems to be related to the AWS module version. I downgraded to an older version (3.37) and it works fine.

1 Like

We tested on 3.52 and it worked. It sounds, like 3.53 introduced a change which is breaking this functionality.

1 Like

Same thing here. We had the exact same issue, and 3.52 works as intended.

This seems to be the matching bug on the aws providers bug tracker: KMS doesn't work with newest verision of provider · Issue #20588 · hashicorp/terraform-provider-aws · GitHub

@renatomrcosta1991 hey, trying to deploy JX3 EKS cluster, following the docs Amazon | Jenkins X - Cloud Native CI/CD Built On Kubernetes
but facing the same issue with Terraform v1.0.7, aws v3.59.0, I Downgraded to v3.52 as suggested, however I am getting the same error.
Can you please help me.
Thanks


│ Error: error waiting for KMS Key (8eee30d8-7c69-476c-8bc0-f6c68f97519e) policy propagation: timeout while waiting for state to become ‘TRUE’ (last state: ‘FALSE’, timeout: 5m0s)

│ with module.eks-jx.module.vault.aws_kms_key.kms_vault_unseal[0],
│ on .terraform/modules/eks-jx/modules/vault/main.tf line 107, in resource “aws_kms_key” “kms_vault_unseal”:

Hi @pradeeprajbhar. In our case changing the provider to anything pre-v3.53 worked as intended. Make sure to run your terraform plan before applying again just in case

Good morning, in my case I have specified the AWS provider version,(3.52), but when i initiate the terraform, by default its going to download the latest provider not the 3.52.

here is my main.tf, .terraform.lock.hcl (with latest aws provider)

main.tf

terraform {
required_providers {
aws = {
source = “hashicorp/aws”
version = “~> 3.52”
}
}
}

Configure the AWS Provider

provider “aws” {
region = “${var.region}”
shared_credentials_file = “/root/.aws/credentials”
profile = “developer”
}

module “eks-jx” {
source = “jenkins-x/eks-jx/aws”
version = “1.15.38”
cluster_version = var.cluster_version
region = var.region
vault_user = var.vault_user
is_jx2 = false
jx_git_url = var.jx_git_url
jx_bot_username = var.jx_bot_username
jx_bot_token = var.jx_bot_token
force_destroy = var.force_destroy
nginx_chart_version = var.nginx_chart_version
install_kuberhealthy = var.install_kuberhealthy
}

=============================

root@:~/ivari-jx3/JX3-codes# terraform --version
Terraform v1.0.7
on linux_amd64

==============
.terraform.lock.hcl
cat .terraform.lock.hcl

This file is maintained automatically by “terraform init”.

Manual edits may be lost in future updates.

provider “registry.terraform.io/hashicorp/aws” {
version = “3.60.0”
constraints = “>= 2.23.0, >= 2.53.0, >= 2.70.0, >= 3.40.0, ~> 3.52, >= 3.56.0, < 4.0.0”
hashes = [
“h1:vwRjnpZOFwDlbFb2WX10JM2zNEEVyRLc8cBwkxCXlAE=”,
“zh:01323eedb8f006c8f9fffdfc23b449625b1446c1e43b8454e4a40a7461193661”,
“zh:03513ffdae205832be480b30d332b47a573e48623390e8f9f833141c8ceccb6a”,
“zh:47611a8b361ced9a3b58b9868be2004677cf4ea0d04cfb5f54c6ae95e997e7c7”,
“zh:9a7e80c2a2ed0f2e59b05e27374daaafd64785161546ed40f4db11048fbc78a7”,
“zh:9e809746c4fdaa4214700e81a67b35f02afc1f2873591b0360c473cfd7193877”,
“zh:a009d48e4ebcf78e24af9299c6a8664e0375411b4f16d5d0d7c7454b12052c10”,
“zh:adc910f48f5ddc402e7653e70429d150d61bee5190aba7495575303aba6ca6c8”,
“zh:b702e219532bc09be58f8a30cb3239626ffc9bc0e42b44497b0644f9ecc657b5”,
“zh:bc50d787593e714acb54d65e8df026490a968e54d2184496efda7ba07c211836”,
“zh:bd74e3b1c815d5a9c710cb5c55f2d5f6742471a23e63f924fd3a6493f384cd43”,
“zh:fa7eb23bcf4c01f93d74c509c0e9b039148f43424c3b4ce64619af17ee12265c”,
]
}

Please have a look, all i wanted to execute the tf with aws provider version 3.52 or 3.53, not sure why its downloading the latest one.
Thanks

Aha @pradeeprajbhar, I think I understand your issue now!

In your main.tf snippet, you defined the version thus:

version = "~> 3.52"

Per the specification (Version Constraints - Configuration Language - Terraform by HashiCorp):

  • ~> : Allows only the rightmost version component to increment. For example, to allow new patch releases within a specific minor release, use the full version number: ~> 1.0.4 will allow installation of 1.0.5 and 1.0.10 but not 1.1.0 . This is usually called the pessimistic constraint operator.

Since your rightmost version number there is the minor, it will still try to resolve to a higher version nonetheless.

In our case, we simple pinned the version without constraint operations:

version = "3.52"

Try planning/applying to see if these changes can help ya. Good luck!

Hi, I made the changes as you suggested, but i am unable to download the modules :frowning:

here the error

root@ip8:~/jx3-terraform-eks# terraform init
Initializing modules…

Initializing the backend…

Initializing provider plugins…

  • Finding hashicorp/kubernetes versions matching “>= 1.11.1, ~> 2.0”…
  • Finding hashicorp/null versions matching “~> 3.0”…
  • Finding hashicorp/local versions matching “>= 1.4.0, ~> 2.0”…
  • Finding hashicorp/random versions matching “~> 3.0”…
  • Finding hashicorp/helm versions matching “~> 2.0”…
  • Finding hashicorp/template versions matching “~> 2.0”…
  • Finding hashicorp/cloudinit versions matching “>= 2.0.0”…
  • Finding terraform-aws-modules/http versions matching “>= 2.4.1”…
  • Finding hashicorp/aws versions matching “>= 2.23.0, >= 2.53.0, >= 2.70.0, >= 3.40.0, 3.52.0, >= 3.56.0, < 4.0.0”…
  • Installing hashicorp/random v3.1.0…
  • Installed hashicorp/random v3.1.0 (signed by HashiCorp)
  • Installing hashicorp/helm v2.3.0…
  • Installed hashicorp/helm v2.3.0 (signed by HashiCorp)
  • Installing hashicorp/template v2.2.0…
  • Installed hashicorp/template v2.2.0 (signed by HashiCorp)
  • Installing hashicorp/cloudinit v2.2.0…
  • Installed hashicorp/cloudinit v2.2.0 (signed by HashiCorp)
  • Installing terraform-aws-modules/http v2.4.1…
  • Installed terraform-aws-modules/http v2.4.1 (self-signed, key ID B2C1C0641B6B0EB7)
  • Installing hashicorp/kubernetes v2.5.0…
  • Installed hashicorp/kubernetes v2.5.0 (signed by HashiCorp)
  • Installing hashicorp/null v3.1.0…
  • Installed hashicorp/null v3.1.0 (signed by HashiCorp)
  • Installing hashicorp/local v2.1.0…
  • Installed hashicorp/local v2.1.0 (signed by HashiCorp)

Partner and community providers are signed by their developers.
If you’d like to know more about provider signing, you can read about it here:


│ Error: Failed to query available provider packages

│ Could not retrieve the list of available versions for provider hashicorp/aws: no available releases match the given constraints >= 2.23.0, >= 2.53.0, >=
│ 2.70.0, >= 3.40.0, 3.52.0, >= 3.56.0, < 4.0.0

1 Like

Hi,
How did you resolve this issue?

Hey, I was curious how you rolled back to that specific version? I tried the 3.52 version that people here are saying is working to see if that is my issue as well, but it won’t let me use anything below 3.56, as per the constraints in my lock file. I tried both updating the lock file itself as well as creating a ‘versions.tf’ file and setting it to the specific version with no luck…

There is a underlying eks module that requires >= 3.56.0.

Also this should now be fixed with a higher version of the aws provider KMS resource Key Policy propagation consistent fail · Issue #21225 · hashicorp/terraform-provider-aws · GitHub

Refer to my issue here for more hints: