Vault failed to unseal core

Hello,

I am using the helm chart v0.7.0 to deploy on Kubernetes running on AWS. From the vault-0 pod I have the following error:

2020-09-23T13:20:56.183Z [INFO]  core: stored unseal keys supported, attempting fetch
2020-09-23T13:20:56.190Z [WARN]  failed to unseal core: error="stored unseal keys are supported, but none were found"
2020-09-23T13:21:00.233Z [INFO]  core.autoseal: seal configuration missing, but cannot check old path as core is sealed: seal_type=recovery

Any ideas about why I’m seeing this error? I have verified that the KMS service works by running aws kms list-keys from another pod.

Additional Info

Here’s the output of vault status:

$ vault status
Key                      Value
---                      -----
Recovery Seal Type       awskms
Initialized              false
Sealed                   true
Total Recovery Shares    0
Threshold                0
Unseal Progress          0/0
Unseal Nonce             n/a
Version                  n/a
HA Enabled               false

And here’s the config from inside the pod:

$ cat /vault/config/extraconfig-from-values.hcl 
disable_mlock = true
ui = true

listener "tcp" {
  tls_disable = 1
  address = "[::]:8200"
  cluster_address = "[::]:8201"
}
storage "s3" {
  bucket = "<S3-BUCKET-REDACTED>"
  region = "eu-west-1"
  path = "/vault/data"
}

seal "awskms" {
  region = "eu-west-1"
  kms_key_id = "<REDACTED>"
  endpoint = "<KMS-VPC-ENDPOINT-REDACTED"
} 

I have the following extraEnvironmentVars and extraSecretEnvironmentVars in my values.yaml:

  extraEnvironmentVars:
    VAULT_SEAL_TYPE: awskms

  extraSecretEnvironmentVars:
    - envName: AWS_REGION
      secretName: vault-aws-auth
      secretKey: AWS_REGION
    - envName: AWS_ACCESS_KEY_ID
      secretName: vault-aws-auth
      secretKey: AWS_ACCESS_KEY_ID
    - envName: AWS_SECRET_ACCESS_KEY
      secretName: vault-aws-auth
      secretKey: AWS_SECRET_ACCESS_KEY
    - envName: VAULT_AWSKMS_SEAL_KEY_ID
      secretName: vault-aws-auth
      secretKey: VAULT_AWSKMS_SEAL_KEY_ID

I would appreciate any help possible. Thank you.

Initialized false

This probably tells what’s wrong. Even when deployed via Helm, you need to run the vault init command to initialize the datastore.

@jeroenjacobs79 Thank you for your suggestion. I followed it and initialized vault but got the following error:

$ vault operator init -key-shares=1 -key-threshold=1 -format=json
Error initializing: Error making API request.

URL: PUT http://127.0.0.1:8200/v1/sys/init
Code: 500. Errors:

* unseal with stored key failed: Vault is not initialized

And now vault status says it is initialized but sealed:

$ vault status
Key                      Value
---                      -----
Recovery Seal Type       shamir
Initialized              true
Sealed                   true
Total Recovery Shares    5
Threshold                3
Unseal Progress          0/3
Unseal Nonce             n/a
Version                  1.5.2
HA Enabled               false

Also, the exit code for vault status is 2 if that means anything.

Hello @jeroenjacobs79
Documentation is not clear.
In order to install Vault via Helm, awskms config must be filled correctly.

seal "awskms" {
        region     = "eu-central-1"
        access_key = "AKIASDFGSTGWHNRYCOIJO"
        secret_key = "5UGVLjVt/CBFx<o78edtD56HDCFASDkhh1Tgi9i8l"
        kms_key_id = "?????????"
        endpoint   = "????????"
      }

but how can we get kms_key_id and endpoint address without/before vault installation?

What is the logical order for installation of Vault on AWS EKS and auto-unseal?

Thanks & Regards