Error waiting for KMS Key - timeout while waiting for state to become 'TRUE'

klydem11 · January 7, 2022, 12:13pm

Hi,

I’ve decided to start fresh with following the EKS Terraform quick start template .
I’ve created the following repos:

jx_cluster
jx_infrastructure
election_web_app

I am having trouble running terraform apply with an error related to the KMS key.
I’ve tried aws prodiver 3.56, 3.69 and 3.70

│ Error: error waiting for KMS Key (fe6c66d0-59ef-4275-b7f8-8d42d5446599) policy propagation: timeout while waiting for state to become 'TRUE' (last state: 'FALSE', timeout: 2m0s)
│
│   with module.eks-jx.module.vault.aws_kms_key.kms_vault_unseal[0],
│   on .terraform/modules/eks-jx/modules/vault/main.tf line 98, in resource "aws_kms_key" "kms_vault_unseal":
│   98: resource "aws_kms_key" "kms_vault_unseal" {

Looking into .terraform/modules/eks-jx/modules/vault/main.tf line 98, in resource “aws_kms_key” “kms_vault_unseal” :

resource "aws_kms_key" "kms_vault_unseal" {

  count = local.create_vault_resources ? 1 : 0

  description         = "KMS Key for bank vault unseal"

  enable_key_rotation = var.enable_key_rotation

  policy              = <<POLICY

{

    "Version": "2012-10-17",

    "Statement": [

        {

            "Sid": "EnableIAMUserPermissions",

            "Effect": "Allow",

            "Principal": {

                "AWS": [

                    "${length(data.aws_iam_user.vault_user) > 0 ? data.aws_iam_user.vault_user[0].arn : ""}",

                    "${data.aws_caller_identity.current.arn}",

                    "arn:${data.aws_partition.current.partition}:iam::${data.aws_caller_identity.current.account_id}:root"

                ]

            },

            "Action": "kms:*",

            "Resource": "*"

        }

    ]

}

POLICY

}

Also I can see a push to the remote jx_cluster repo:

The contents of variable.tf being:

// ----------------------------------------------------------------------------
// Optional Variables
// ----------------------------------------------------------------------------
variable "region" {
  description = "AWS region code for creating resources."
  type        = string
  default     = "eu-west-2"
}

variable "profile" {
  description = "Profile stored in aws config or credentials file"
  type        = string
  default     = "default"
}

variable "cluster_version" {
  description = "Kubernetes version to use for the EKS cluster."
  type        = string
  default     =  "1.21"
}

variable "vault_user" {
  description = "The AWS IAM Username whose credentials will be used to authenticate the Vault pods against AWS"
  type        = string
  default     = ""
}

variable "cluster_name" {
  description = "Name of the Kubernetes cluster to create"
  type        = string
  default     = "jx_k8_cluster"
}

variable "force_destroy" {
  description = "Flag to determine whether storage buckets get forcefully destroyed. If set to false, empty the bucket first in the aws s3 console, else terraform destroy will fail with BucketNotEmpty error"
  type        = bool
  default     = true
}

variable "is_jx2" {
  default     = false
  type        = bool
  description = "Flag to specify if jx2 related resources need to be created"
}

variable "jx_git_url" {
  description = "URL for the Jenins X cluster git repository"
  type        = string
  default     = "https://github.com/surreyfyp/jx_cluster.git"
}

variable "jx_bot_username" {
  description = "Bot username used to interact with the Jenkins X cluster git repository"
  type        = string
  default     = "xxxxx"
}

variable "jx_bot_token" {
  description = "Bot token used to interact with the Jenkins X cluster git repository"
  type        = string
  default     = "xxxxx"
}

variable "nginx_chart_version" {
  type        = string
  description = "nginx chart version"
  default     = "3.12.0"
}

variable "install_kuberhealthy" {
  description = "Flag to specify if kuberhealthy operator should be installed"
  type        = bool
  default     = true
}

The KMS Key error with terraform DEBUG:

module.eks-jx.module.vault.aws_kms_key.kms_vault_unseal[0]: Still creating... [1m50s elapsed]
2022-01-06T14:27:00.288Z [INFO]  provider.terraform-provider-aws_v3.56.0_x5: 2022/01/06 14:27:00 [DEBUG] [aws-sdk-go] DEBUG: Request kms/GetKeyPolicy Details:
---[ REQUEST POST-SIGN ]-----------------------------
POST / HTTP/1.1
Host: kms.eu-west-2.amazonaws.com
User-Agent: APN/1.0 HashiCorp/1.0 Terraform/1.1.2 (+https://www.terraform.io) terraform-provider-aws/3.56.0 (+https://registry.terraform.io/providers/hashicorp/aws) aws-sdk-go/1.40.29 (go1.16; linux; amd64)
Content-Length: 71
Authorization: AWS4-HMAC-SHA256 Credential=xxx/20220106/eu-west-2/kms/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-date;x-amz-target, Signature=6606b7439775bbc104c94f8a742b4db50e781ae64c7f2ffb54ccfe732fac0da8
Content-Type: application/x-amz-json-1.1
X-Amz-Date: 20220106T142700Z
X-Amz-Target: TrentService.GetKeyPolicy
Accept-Encoding: gzip

{"KeyId":"fe6c66d0-59ef-4275-b7f8-8d42d5446599","PolicyName":"default"}
-----------------------------------------------------: timestamp=2022-01-06T14:27:00.284Z
2022-01-06T14:27:00.329Z [INFO]  provider.terraform-provider-aws_v3.56.0_x5: 2022/01/06 14:27:00 [DEBUG] [aws-sdk-go] DEBUG: Response kms/GetKeyPolicy Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
Connection: close
Content-Length: 348
Cache-Control: no-cache, no-store, must-revalidate, private
Content-Type: application/x-amz-json-1.1
Date: Thu, 06 Jan 2022 14:26:59 GMT
Expires: 0
Pragma: no-cache
X-Amzn-Requestid: 81a3d944-87e9-49a9-b1fb-6f98cbaac2b0


-----------------------------------------------------: timestamp=2022-01-06T14:27:00.328Z
2022-01-06T14:27:00.329Z [INFO]  provider.terraform-provider-aws_v3.56.0_x5: 2022/01/06 14:27:00 [DEBUG] [aws-sdk-go] {"Policy":"{\n  \"Version\" : \"2012-10-17\",\n  \"Statement\" : [ {\n    \"Sid\" : \"EnableIAMUserPermissions\",\n    \"Effect\" : \"Allow\",\n    \"Principal\" : {\n      \"AWS\" : [ \"arn:aws:iam::847399026905:root\", \"arn:aws:iam::847399026905:user/jenkins-x-vault\" ]\n    },\n    \"Action\" : \"kms:*\",\n    \"Resource\" : \"*\"\n  } ]\n}"}: timestamp=2022-01-06T14:27:00.329Z
2022-01-06T14:27:00.329Z [INFO]  provider.terraform-provider-aws_v3.56.0_x5: 2022/01/06 14:27:00 [TRACE] Waiting 10s before next try: timestamp=2022-01-06T14:27:00.329Z
Key ARN(s):

The KMS Key error Logs with export TF_LOG_CORE=TRACE and export TF_LOG_PROVIDER=TRACE:

module.eks-jx.module.vault.aws_kms_key.kms_vault_unseal[0]: Still creating... [4m50s elapsed]
2022-01-06T23:57:19.229Z [INFO]  provider.terraform-provider-aws_v3.69.0_x5: 2022/01/06 23:57:19 [TRACE] Waiting 10s before next try: timestamp=2022-01-06T23:57:19.228Z
2022-01-06T23:57:20.100Z [TRACE] dag/walk: vertex "module.eks-jx.module.vault.aws_iam_user_policy_attachment.attach_vault_policy_to_user[0]" is waiting for "module.eks-jx.module.vault.aws_iam_policy.aws_vault_user_policy[0]"
2022-01-06T23:57:20.100Z [TRACE] dag/walk: vertex "module.eks-jx.local.compact_content (expand)" is waiting for "module.eks-jx.local.split_content (expand)"
2022-01-06T23:57:20.100Z [TRACE] dag/walk: vertex "module.eks-jx.local.split_content (expand)" is waiting for "module.eks-jx.local.interpolated_content (expand)"
2022-01-06T23:57:20.100Z [TRACE] dag/walk: vertex "module.eks-jx.module.cluster.kubernetes_config_map.jenkins_x_requirements[0]" is waiting for "module.eks-jx.module.cluster.var.content (expand)"
2022-01-06T23:57:20.101Z [TRACE] dag/walk: vertex "module.eks-jx.module.vault.output.kms_vault_unseal (expand)" is waiting for "module.eks-jx.module.vault.aws_kms_key.kms_vault_unseal[0]"
2022-01-06T23:57:20.101Z [TRACE] dag/walk: vertex "module.eks-jx.local.content (expand)" is waiting for "module.eks-jx.local.compact_content (expand)"
2022-01-06T23:57:20.101Z [TRACE] dag/walk: vertex "module.eks-jx.output.vault_kms_unseal (expand)" is waiting for "module.eks-jx.module.vault.output.kms_vault_unseal (expand)"
2022-01-06T23:57:20.102Z [TRACE] dag/walk: vertex "output.vault_kms_unseal" is waiting for "module.eks-jx.output.vault_kms_unseal (expand)"
2022-01-06T23:57:20.102Z [TRACE] dag/walk: vertex "module.eks-jx.module.cluster.provider[\"registry.terraform.io/hashicorp/kubernetes\"] (close)" is waiting for "module.eks-jx.module.cluster.kubernetes_config_map.jenkins_x_requirements[0]"
2022-01-06T23:57:20.102Z [TRACE] dag/walk: vertex "module.eks-jx.module.cluster.var.content (expand)" is waiting for "module.eks-jx.local.content (expand)"
2022-01-06T23:57:20.102Z [TRACE] dag/walk: vertex "root" is waiting for "output.vault_kms_unseal"
2022-01-06T23:57:20.613Z [TRACE] dag/walk: vertex "module.eks-jx.module.vault.data.aws_iam_policy_document.vault_iam_user_policy_document[0]" is waiting for "module.eks-jx.module.vault.aws_kms_key.kms_vault_unseal[0]"
2022-01-06T23:57:20.613Z [TRACE] dag/walk: vertex "module.eks-jx.module.vault.aws_iam_policy.aws_vault_user_policy[0]" is waiting for "module.eks-jx.module.vault.data.aws_iam_policy_document.vault_iam_user_policy_document[0]"
2022-01-06T23:57:20.613Z [TRACE] dag/walk: vertex "module.eks-jx.module.vault (close)" is waiting for "module.eks-jx.module.vault.output.kms_vault_unseal (expand)"
2022-01-06T23:57:20.618Z [TRACE] dag/walk: vertex "module.eks-jx.local.interpolated_content (expand)" is waiting for "module.eks-jx.module.vault.output.kms_vault_unseal (expand)"
2022-01-06T23:57:20.621Z [TRACE] dag/walk: vertex "module.eks-jx.module.cluster (close)" is waiting for "module.eks-jx.module.cluster.kubernetes_config_map.jenkins_x_requirements[0]"
2022-01-06T23:57:20.632Z [TRACE] dag/walk: vertex "provider[\"registry.terraform.io/hashicorp/aws\"] (close)" is waiting for "module.eks-jx.module.vault.aws_iam_user_policy_attachment.attach_vault_policy_to_user[0]"
2022-01-06T23:57:22.973Z [TRACE] dag/walk: vertex "module.eks-jx (close)" is waiting for "module.eks-jx.output.vault_kms_unseal (expand)"
2022-01-06T23:57:25.101Z [TRACE] dag/walk: vertex "module.eks-jx.module.vault.aws_iam_user_policy_attachment.attach_vault_policy_to_user[0]" is waiting for "module.eks-jx.module.vault.aws_iam_policy.aws_vault_user_policy[0]"
2022-01-06T23:57:25.101Z [TRACE] dag/walk: vertex "module.eks-jx.local.compact_content (expand)" is waiting for "module.eks-jx.local.split_content (expand)"
2022-01-06T23:57:25.102Z [TRACE] dag/walk: vertex "module.eks-jx.local.content (expand)" is waiting for "module.eks-jx.local.compact_content (expand)"
2022-01-06T23:57:25.102Z [TRACE] dag/walk: vertex "module.eks-jx.local.split_content (expand)" is waiting for "module.eks-jx.local.interpolated_content (expand)"
2022-01-06T23:57:25.102Z [TRACE] dag/walk: vertex "module.eks-jx.module.cluster.kubernetes_config_map.jenkins_x_requirements[0]" is waiting for "module.eks-jx.module.cluster.var.content (expand)"
2022-01-06T23:57:25.102Z [TRACE] dag/walk: vertex "module.eks-jx.module.vault.output.kms_vault_unseal (expand)" is waiting for "module.eks-jx.module.vault.aws_kms_key.kms_vault_unseal[0]"
2022-01-06T23:57:25.102Z [TRACE] dag/walk: vertex "module.eks-jx.output.vault_kms_unseal (expand)" is waiting for "module.eks-jx.module.vault.output.kms_vault_unseal (expand)"
2022-01-06T23:57:25.102Z [TRACE] dag/walk: vertex "output.vault_kms_unseal" is waiting for "module.eks-jx.output.vault_kms_unseal (expand)"
2022-01-06T23:57:25.103Z [TRACE] dag/walk: vertex "root" is waiting for "output.vault_kms_unseal"
2022-01-06T23:57:25.103Z [TRACE] dag/walk: vertex "module.eks-jx.module.cluster.var.content (expand)" is waiting for "module.eks-jx.local.content (expand)"
2022-01-06T23:57:25.103Z [TRACE] dag/walk: vertex "module.eks-jx.module.cluster.provider[\"registry.terraform.io/hashicorp/kubernetes\"] (close)" is waiting for "module.eks-jx.module.cluster.kubernetes_config_map.jenkins_x_requirements[0]"
2022-01-06T23:57:25.614Z [TRACE] dag/walk: vertex "module.eks-jx.module.vault.data.aws_iam_policy_document.vault_iam_user_policy_document[0]" is waiting for "module.eks-jx.module.vault.aws_kms_key.kms_vault_unseal[0]"
2022-01-06T23:57:25.614Z [TRACE] dag/walk: vertex "module.eks-jx.module.vault (close)" is waiting for "module.eks-jx.module.vault.output.kms_vault_unseal (expand)"
2022-01-06T23:57:25.614Z [TRACE] dag/walk: vertex "module.eks-jx.module.vault.aws_iam_policy.aws_vault_user_policy[0]" is waiting for "module.eks-jx.module.vault.data.aws_iam_policy_document.vault_iam_user_policy_document[0]"
2022-01-06T23:57:25.619Z [TRACE] dag/walk: vertex "module.eks-jx.local.interpolated_content (expand)" is waiting for "module.eks-jx.module.vault.output.kms_vault_unseal (expand)"
2022-01-06T23:57:25.622Z [TRACE] dag/walk: vertex "module.eks-jx.module.cluster (close)" is waiting for "module.eks-jx.module.cluster.kubernetes_config_map.jenkins_x_requirements[0]"
2022-01-06T23:57:25.632Z [TRACE] dag/walk: vertex "provider[\"registry.terraform.io/hashicorp/aws\"] (close)" is waiting for "module.eks-jx.module.vault.aws_iam_user_policy_attachment.attach_vault_policy_to_user[0]"
2022-01-06T23:57:27.976Z [TRACE] dag/walk: vertex "module.eks-jx (close)" is waiting for "module.eks-jx.output.vault_kms_unseal (expand)"
module.eks-jx.module.vault.aws_kms_key.kms_vault_unseal[0]: Still creating... [5m0s elapsed]
2022-01-06T23:57:29.298Z [INFO]  provider.terraform-provider-aws_v3.69.0_x5: 2022/01/06 23:57:29 [TRACE] Waiting 10s before next try: timestamp=2022-01-06T23:57:29.298Z
2022-01-06T23:57:30.102Z [TRACE] dag/walk: vertex "module.eks-jx.module.vault.aws_iam_user_policy_attachment.attach_vault_policy_to_user[0]" is waiting for "module.eks-jx.module.vault.aws_iam_policy.aws_vault_user_policy[0]"
2022-01-06T23:57:30.102Z [TRACE] dag/walk: vertex "module.eks-jx.local.compact_content (expand)" is waiting for "module.eks-jx.local.split_content (expand)"
2022-01-06T23:57:30.102Z [TRACE] dag/walk: vertex "module.eks-jx.module.vault.output.kms_vault_unseal (expand)" is waiting for "module.eks-jx.module.vault.aws_kms_key.kms_vault_unseal[0]"
2022-01-06T23:57:30.102Z [TRACE] dag/walk: vertex "module.eks-jx.local.split_content (expand)" is waiting for "module.eks-jx.local.interpolated_content (expand)"
2022-01-06T23:57:30.102Z [TRACE] dag/walk: vertex "module.eks-jx.local.content (expand)" is waiting for "module.eks-jx.local.compact_content (expand)"
2022-01-06T23:57:30.103Z [TRACE] dag/walk: vertex "output.vault_kms_unseal" is waiting for "module.eks-jx.output.vault_kms_unseal (expand)"
2022-01-06T23:57:30.103Z [TRACE] dag/walk: vertex "module.eks-jx.output.vault_kms_unseal (expand)" is waiting for "module.eks-jx.module.vault.output.kms_vault_unseal (expand)"
2022-01-06T23:57:30.103Z [TRACE] dag/walk: vertex "module.eks-jx.module.cluster.kubernetes_config_map.jenkins_x_requirements[0]" is waiting for "module.eks-jx.module.cluster.var.content (expand)"
2022-01-06T23:57:30.103Z [TRACE] dag/walk: vertex "root" is waiting for "output.vault_kms_unseal"
2022-01-06T23:57:30.103Z [TRACE] dag/walk: vertex "module.eks-jx.module.cluster.var.content (expand)" is waiting for "module.eks-jx.local.content (expand)"
2022-01-06T23:57:30.103Z [TRACE] dag/walk: vertex "module.eks-jx.module.cluster.provider[\"registry.terraform.io/hashicorp/kubernetes\"] (close)" is waiting for "module.eks-jx.module.cluster.kubernetes_config_map.jenkins_x_requirements[0]"
2022-01-06T23:57:30.614Z [TRACE] dag/walk: vertex "module.eks-jx.module.vault.data.aws_iam_policy_document.vault_iam_user_policy_document[0]" is waiting for "module.eks-jx.module.vault.aws_kms_key.kms_vault_unseal[0]"
2022-01-06T23:57:30.614Z [TRACE] dag/walk: vertex "module.eks-jx.module.vault.aws_iam_policy.aws_vault_user_policy[0]" is waiting for "module.eks-jx.module.vault.data.aws_iam_policy_document.vault_iam_user_policy_document[0]"
2022-01-06T23:57:30.614Z [TRACE] dag/walk: vertex "module.eks-jx.module.vault (close)" is waiting for "module.eks-jx.module.vault.output.kms_vault_unseal (expand)"
2022-01-06T23:57:30.619Z [TRACE] dag/walk: vertex "module.eks-jx.local.interpolated_content (expand)" is waiting for "module.eks-jx.module.vault.output.kms_vault_unseal (expand)"
2022-01-06T23:57:30.622Z [TRACE] dag/walk: vertex "module.eks-jx.module.cluster (close)" is waiting for "module.eks-jx.module.cluster.kubernetes_config_map.jenkins_x_requirements[0]"
2022-01-06T23:57:30.633Z [TRACE] dag/walk: vertex "provider[\"registry.terraform.io/hashicorp/aws\"] (close)" is waiting for "module.eks-jx.module.vault.aws_iam_user_policy_attachment.attach_vault_policy_to_user[0]"
2022-01-06T23:57:32.753Z [INFO]  provider.terraform-provider-aws_v3.69.0_x5: 2022/01/06 23:57:32 [WARN] WaitForState timeout after 5m0s: timestamp=2022-01-06T23:57:32.752Z
2022-01-06T23:57:32.753Z [INFO]  provider.terraform-provider-aws_v3.69.0_x5: 2022/01/06 23:57:32 [WARN] WaitForState starting 30s refresh grace period: timestamp=2022-01-06T23:57:32.753Z
2022-01-06T23:57:32.756Z [TRACE] maybeTainted: module.eks-jx.module.vault.aws_kms_key.kms_vault_unseal[0] encountered an error during creation, so it is now marked as tainted
2022-01-06T23:57:32.756Z [TRACE] NodeAbstractResouceInstance.writeResourceInstanceState to workingState for module.eks-jx.module.vault.aws_kms_key.kms_vault_unseal[0]
2022-01-06T23:57:32.756Z [TRACE] NodeAbstractResouceInstance.writeResourceInstanceState: writing state object for module.eks-jx.module.vault.aws_kms_key.kms_vault_unseal[0]
2022-01-06T23:57:32.756Z [TRACE] evalApplyProvisioners: module.eks-jx.module.vault.aws_kms_key.kms_vault_unseal[0] is tainted, so skipping provisioning
2022-01-06T23:57:32.756Z [TRACE] maybeTainted: module.eks-jx.module.vault.aws_kms_key.kms_vault_unseal[0] was already tainted, so nothing to do
2022-01-06T23:57:32.756Z [TRACE] NodeAbstractResouceInstance.writeResourceInstanceState to workingState for module.eks-jx.module.vault.aws_kms_key.kms_vault_unseal[0]
2022-01-06T23:57:32.757Z [TRACE] NodeAbstractResouceInstance.writeResourceInstanceState: writing state object for module.eks-jx.module.vault.aws_kms_key.kms_vault_unseal[0]
2022-01-06T23:57:32.758Z [TRACE] statemgr.Filesystem: have already backed up original terraform.tfstate to terraform.tfstate.backup on a previous write
2022-01-06T23:57:32.772Z [TRACE] statemgr.Filesystem: state has changed since last snapshot, so incrementing serial to 117
2022-01-06T23:57:32.772Z [TRACE] statemgr.Filesystem: writing snapshot at terraform.tfstate
2022-01-06T23:57:32.976Z [TRACE] dag/walk: vertex "module.eks-jx (close)" is waiting for "module.eks-jx.output.vault_kms_unseal (expand)"
2022-01-06T23:57:33.110Z [ERROR] vertex "module.eks-jx.module.vault.aws_kms_key.kms_vault_unseal[0]" error: error waiting for KMS Key (9c0be3d7-5f76-40b6-a775-e9b9628a15b9) policy propagation: timeout while waiting for state to become 'TRUE' (last state: 'FALSE', timeout: 5m0s)
2022-01-06T23:57:33.110Z [TRACE] vertex "module.eks-jx.module.vault.aws_kms_key.kms_vault_unseal[0]": visit complete, with errors
2022-01-06T23:57:33.110Z [TRACE] dag/walk: upstream of "module.eks-jx.module.vault.output.kms_vault_unseal (expand)" errored, so skipping
2022-01-06T23:57:33.110Z [TRACE] dag/walk: upstream of "module.eks-jx.output.vault_kms_unseal (expand)" errored, so skipping
2022-01-06T23:57:33.110Z [TRACE] dag/walk: upstream of "output.vault_kms_unseal" errored, so skipping
2022-01-06T23:57:33.110Z [TRACE] dag/walk: upstream of "module.eks-jx.local.interpolated_content (expand)" errored, so skipping
2022-01-06T23:57:33.111Z [TRACE] dag/walk: upstream of "module.eks-jx.local.split_content (expand)" errored, so skipping
2022-01-06T23:57:33.111Z [TRACE] dag/walk: upstream of "module.eks-jx.local.compact_content (expand)" errored, so skipping
2022-01-06T23:57:33.111Z [TRACE] dag/walk: upstream of "module.eks-jx.module.vault.data.aws_iam_policy_document.vault_iam_user_policy_document[0]" errored, so skipping
2022-01-06T23:57:33.111Z [TRACE] dag/walk: upstream of "module.eks-jx.local.content (expand)" errored, so skipping
2022-01-06T23:57:33.111Z [TRACE] dag/walk: upstream of "module.eks-jx.module.cluster.var.content (expand)" errored, so skipping
2022-01-06T23:57:33.111Z [TRACE] dag/walk: upstream of "module.eks-jx.module.cluster.kubernetes_config_map.jenkins_x_requirements[0]" errored, so skipping
2022-01-06T23:57:33.111Z [TRACE] dag/walk: upstream of "module.eks-jx.module.cluster.provider[\"registry.terraform.io/hashicorp/kubernetes\"] (close)" errored, so skipping
2022-01-06T23:57:33.111Z [TRACE] dag/walk: upstream of "module.eks-jx.module.vault.aws_iam_policy.aws_vault_user_policy[0]" errored, so skipping
2022-01-06T23:57:33.111Z [TRACE] dag/walk: upstream of "module.eks-jx.module.vault.aws_iam_user_policy_attachment.attach_vault_policy_to_user[0]" errored, so skipping
2022-01-06T23:57:33.111Z [TRACE] dag/walk: upstream of "provider[\"registry.terraform.io/hashicorp/aws\"] (close)" errored, so skipping
2022-01-06T23:57:33.111Z [TRACE] dag/walk: upstream of "module.eks-jx.module.cluster (close)" errored, so skipping
2022-01-06T23:57:33.111Z [TRACE] dag/walk: upstream of "module.eks-jx.module.vault (close)" errored, so skipping
2022-01-06T23:57:33.111Z [TRACE] dag/walk: upstream of "module.eks-jx (close)" errored, so skipping
2022-01-06T23:57:33.111Z [TRACE] dag/walk: upstream of "root" errored, so skipping
2022-01-06T23:57:33.111Z [TRACE] statemgr.Filesystem: have already backed up original terraform.tfstate to terraform.tfstate.backup on a previous write
2022-01-06T23:57:33.123Z [TRACE] statemgr.Filesystem: state has changed since last snapshot, so incrementing serial to 118
2022-01-06T23:57:33.123Z [TRACE] statemgr.Filesystem: writing snapshot at terraform.tfstate
╷
│ Error: error waiting for KMS Key (9c0be3d7-5f76-40b6-a775-e9b9628a15b9) policy propagation: timeout while waiting for state to become 'TRUE' (last state: 'FALSE', timeout: 5m0s)
│
│   with module.eks-jx.module.vault.aws_kms_key.kms_vault_unseal[0],
│   on .terraform/modules/eks-jx/modules/vault/main.tf line 98, in resource "aws_kms_key" "kms_vault_unseal":
│   98: resource "aws_kms_key" "kms_vault_unseal" {
│
╵

thomas.thiemann · February 7, 2022, 4:52pm

Same issues here.
I’ve tried some other aws provider versions as the thread starter:
3.69, 3.70, 3.71 and 3.74

module.eks-jx.module.vault.aws_kms_key.kms_vault_unseal[0]: Still creating… [5m10s elapsed]
╷
│ Error: error waiting for KMS Key (ccdef861-6116-4803-804f-600db99f8948) policy propagation: timeout while waiting for state to become ‘TRUE’ (last state: ‘FALSE’, timeout: 5m0s)
│
│ with module.eks-jx.module.vault.aws_kms_key.kms_vault_unseal[0],
│ on .terraform/modules/eks-jx/modules/vault/main.tf line 98, in resource “aws_kms_key” “kms_vault_unseal”:
│ 98: resource “aws_kms_key” “kms_vault_unseal” {

Checked on AWS: The kms key is created and a policy is attached:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "EnableIAMUserPermissions",
            "Effect": "Allow",
            "Principal": {
                "AWS": [
                    "arn:aws:iam::xxxxxxxxxx:user/jenkins-x-vault",
                    "arn:aws:iam::xxxxxxxxxx:root"
                ]
            },
            "Action": "kms:*",
            "Resource": "*"
        }
    ]
}

klydem11 · February 7, 2022, 5:50pm

Hi,

It was happening because i did not configure the aws cli to use a named profile. I had it configured to use the root user. Using the Admin user fixed this error for me.

If your trying to use jenkins x and using the admin user doesn’t resolve it; you can refering to my issue here:

github.com/jenkins-x/jx

error waiting for KMS Key - timeout while waiting for state to become 'TRUE'

opened 03:27PM - 10 Dec 21 UTC

closed 09:25PM - 28 Dec 21 UTC

klydem11

Receiving the following error after attempting to follow the [EKS Terraform quic…k start template](https://github.com/jx3-gitops-repositories/jx3-terraform-eks) ``` │ Error: error waiting for KMS Key (eea6239e-40cc-408a-a412-fa1ece308c15) policy propagation: timeout while waiting for state to become 'TRUE' (last state: 'FALSE', timeout: 5m0s) │ │ with module.eks-jx.module.vault.aws_kms_key.kms_vault_unseal[0], │ on .terraform\modules\eks-jx\modules\vault\main.tf line 98, in resource "aws_kms_key" "kms_vault_unseal": │ 98: resource "aws_kms_key" "kms_vault_unseal" { ``` I've seen the resolution to this in this thread [here](https://github.com/hashicorp/terraform-provider-aws/issues/20588), which says I should downgrade the aws provider to 3.52. Although upon attempting to do that I get the following error regarding constraints. ``` │ Error: Failed to query available provider packages │ │ Could not retrieve the list of available versions for provider │ hashicorp/aws: locked provider registry.terraform.io/hashicorp/aws 3.52.0 │ does not match configured version constraint 1.52.0, >= 2.23.0, >= 2.53.0, │ >= 2.70.0, >= 3.56.0, < 4.0.0; must use terraform init -upgrade to allow │ selection of new versions ``` Terraform Version `v1.1.0` Registry Versions ``` + provider registry.terraform.io/hashicorp/aws v3.69.0 + provider registry.terraform.io/hashicorp/cloudinit v2.2.0 + provider registry.terraform.io/hashicorp/helm v2.4.1 + provider registry.terraform.io/hashicorp/kubernetes v2.7.1 + provider registry.terraform.io/hashicorp/local v2.1.0 + provider registry.terraform.io/hashicorp/null v3.1.0 + provider registry.terraform.io/hashicorp/random v3.1.0 + provider registry.terraform.io/hashicorp/template v2.2.0 + provider registry.terraform.io/terraform-aws-modules/http v2.4.1 ``` My Variables.tf File contains: ``` // ---------------------------------------------------------------------------- // Optional Variables // ---------------------------------------------------------------------------- variable "region" { description = "AWS region code for creating resources." type = string default = "eu-west-2" } variable "profile" { description = "Profile stored in aws config or credentials file" type = string default = "default" } variable "cluster_version" { description = "Kubernetes version to use for the EKS cluster." type = string default = "1.21" } variable "vault_user" { description = "The AWS IAM Username whose credentials will be used to authenticate the Vault pods against AWS" type = string default = "Administrator" } variable "cluster_name" { description = "Name of the Kubernetes cluster to create" type = string default = "jx-infrastructure" } variable "force_destroy" { description = "Flag to determine whether storage buckets get forcefully destroyed. If set to false, empty the bucket first in the aws s3 console, else terraform destroy will fail with BucketNotEmpty error" type = bool default = false } variable "is_jx2" { default = false type = bool description = "Flag to specify if jx2 related resources need to be created" } variable "jx_git_url" { description = "URL for the Jenins X cluster git repository" type = string default = "" // I've set to my Jenins X cluster git repo } variable "jx_bot_username" { description = "Bot username used to interact with the Jenkins X cluster git repository" type = string default = ""// I've set the default to bot username } variable "jx_bot_token" { description = "Bot token used to interact with the Jenkins X cluster git repository" type = string default = "" // I've set the default to my Bot Token } variable "nginx_chart_version" { type = string description = "nginx chart version" default = "3.12.0" } variable "install_kuberhealthy" { description = "Flag to specify if kuberhealthy operator should be installed" type = bool default = true } ```

Hopefully it would be helpful

alekspickle · February 11, 2022, 4:30pm

Did you ever found out what exactly is wrong on terraform side? Because I am having this outside of jenkins context and with ... description propagation: error for KMS key. I do not set policy explicitly either.

Error: error waiting for KMS Key (_) description propagation: timeout while waiting for state to become 'TRUE' (last state: 'FALSE', timeout: 5m0

trying to upgrade to 3.70 because of some fixed team did there.

credibleforce · January 30, 2024, 4:19am

Had this same issue, but only when the user executing terraform was the root user in AWS. After reviewing the key policy I realized that I was already adding the root user to the policy AND then adding the current user. When both users were in the policy I would see only one on the AWS side but the KMS policy propagation would never complete. Added a check to see if current user is root user and not add a duplicate and the issue was resolved.

Topic		Replies	Views
Error waiting for KMS Key (terraform-aws-provider 3.70) AWS tf-aws-provider-release	0	616	February 11, 2022
AWS KMS policy not updating properly Terraform Providers	0	15	October 18, 2024
AWS KMS terraform module not working on apply stage AWS	1	117	August 8, 2024
Unseal and init vault on AWS using awskms - not booting/starting Vault k8s , vault	2	1171	August 24, 2022
Error: waiting for EKS Add-On (education-eks-MIshm4fz:aws-ebs-csi-driver) create: timeout AWS Cloud Control	1	378	March 8, 2024

Error waiting for KMS Key - timeout while waiting for state to become 'TRUE'

Related topics