I have deployed Vault on a GKE cluster(asia-south) and have set up the service account correctly. I am also trying to use GCP KMS(Asia region).
When I try to run “vault operator init” from the vault-0 pod (3 pods in total), it executes the commands connected properly to KMS, but it ends up throwing “Error initializing: context deadline exceeded”. However, as per the vault logs, the setup has been initialed.
When I try without the GCP KMS, it prints out the initial Unseal tokens along with the root token.
The problem now is that now I don’t have the root token/unseal tokens to the work with vault even though the vault is properly initialized. Please help
“context deadline reached” == “time out” in go in most situations.
You didn’t provide any configuration but my guess is that your pod doesn’t have access to the egress or the egress can’t reach GCP’s KMS url. In AWS there is a default region if you don’t set one, do you need one for GCP?
Thanks for a quick reply. So the config is something like this
seal “gcpckms” {
project = “”
region = “asia”
key_ring = “vault”
crypto_key = “vault-key”
}
I can assure you that I can reach the KMS Url because if if I provide a invalid key_ring and crypto_key then it says that the “not found”. Is there a way of extending the context deadline ?
2022-07-11T06:18:42.417Z [INFO] core: security barrier not initialized
2022-07-11T06:18:42.515Z [INFO] core.autoseal: seal configuration missing, but cannot check old path as core is sealed: seal_type=recovery
2022-07-11T06:18:46.455Z [INFO] core: stored unseal keys supported, attempting fetch
2022-07-11T06:18:46.539Z [WARN] failed to unseal core: error="stored unseal keys are supported, but none were found"
2022-07-11T06:18:47.424Z [INFO] core: security barrier not initialized
2022-07-11T06:18:47.499Z [INFO] core.autoseal: seal configuration missing, but cannot check old path as core is sealed: seal_type=recovery
2022-07-11T06:18:51.540Z [INFO] core: stored unseal keys supported, attempting fetch
2022-07-11T06:18:51.623Z [WARN] failed to unseal core: error="stored unseal keys are supported, but none were found"
2022-07-11T06:18:52.724Z [INFO] core: security barrier not initialized
2022-07-11T06:18:53.104Z [INFO] core.autoseal: seal configuration missing, but cannot check old path as core is sealed: seal_type=recovery
2022-07-11T06:18:56.624Z [INFO] core: stored unseal keys supported, attempting fetch
2022-07-11T06:18:56.709Z [WARN] failed to unseal core: error="stored unseal keys are supported, but none were found"
2022-07-11T06:18:57.646Z [INFO] core: security barrier not initialized
2022-07-11T06:18:57.752Z [INFO] core.autoseal: seal configuration missing, but cannot check old path as core is sealed: seal_type=recovery
2022-07-11T06:19:01.709Z [INFO] core: stored unseal keys supported, attempting fetch
2022-07-11T06:19:01.790Z [WARN] failed to unseal core: error="stored unseal keys are supported, but none were found"
2022-07-11T06:19:02.648Z [INFO] core: security barrier not initialized
2022-07-11T06:19:02.729Z [INFO] core.autoseal: seal configuration missing, but cannot check old path as core is sealed: seal_type=recovery
2022-07-11T06:19:03.604Z [WARN] core: stored keys supported on init, forcing shares/threshold to 1
2022-07-11T06:19:03.987Z [INFO] core: security barrier not initialized
2022-07-11T06:19:04.995Z [INFO] core: security barrier initialized: stored=1 shares=1 threshold=1
2022-07-11T06:19:06.343Z [INFO] core: post-unseal setup starting
2022-07-11T06:19:07.537Z [INFO] core: loaded wrapping token key
2022-07-11T06:19:07.921Z [INFO] core.autoseal: seal configuration missing, but cannot check old path as core is sealed: seal_type=recovery
2022-07-11T06:19:08.247Z [INFO] core: Recorded vault version: vault version=1.10.3 upgrade time="2022-07-11 06:19:07.53800754 +0000 UTC"
2022-07-11T06:19:08.324Z [INFO] core: successfully setup plugin catalog: plugin-directory=""
2022-07-11T06:19:08.709Z [INFO] core: no mounts; adding default mount table
2022-07-11T06:19:09.334Z [INFO] core: successfully mounted backend: type=cubbyhole path=cubbyhole/
2022-07-11T06:19:09.335Z [INFO] core: successfully mounted backend: type=system path=sys/
2022-07-11T06:19:09.335Z [INFO] core: successfully mounted backend: type=identity path=identity/
2022-07-11T06:19:12.257Z [INFO] core.autoseal: seal configuration missing, but cannot check old path as core is sealed: seal_type=recovery
2022-07-11T06:19:12.609Z [INFO] core: successfully enabled credential backend: type=token path=token/ namespace="ID: root. Path: "
2022-07-11T06:19:13.258Z [INFO] rollback: starting rollback manager
2022-07-11T06:19:13.258Z [INFO] core: restoring leases
2022-07-11T06:19:13.347Z [INFO] expiration: lease restore complete
2022-07-11T06:19:14.112Z [INFO] identity: entities restored
2022-07-11T06:19:14.194Z [INFO] identity: groups restored
2022-07-11T06:19:15.720Z [WARN] core: post-unseal upgrade seal keys failed: error="no recovery key found"
2022-07-11T06:19:15.720Z [INFO] core: usage gauge collection is disabled
2022-07-11T06:19:17.258Z [INFO] core.autoseal: seal configuration missing, but cannot check old path as core is sealed: seal_type=recovery
2022-07-11T06:19:18.681Z [INFO] core: post-unseal setup complete
2022-07-11T06:19:20.641Z [INFO] core: root token generated
2022-07-11T06:19:20.641Z [INFO] core: pre-seal teardown starting
2022-07-11T06:19:20.641Z [INFO] rollback: stopping rollback manager
2022-07-11T06:19:20.641Z [INFO] core: pre-seal teardown complete
2022-07-11T06:19:50.669Z [INFO] core: stored unseal keys supported, attempting fetch
2022-07-11T06:19:51.081Z [INFO] core.cluster-listener.tcp: starting listener: listener_address=[::]:8201
2022-07-11T06:19:51.082Z [INFO] core.cluster-listener: serving cluster requests: cluster_listen_address=[::]:8201
2022-07-11T06:19:51.082Z [INFO] core: vault is unsealed
2022-07-11T06:19:51.082Z [INFO] core: entering standby mode
2022-07-11T06:20:51.128Z [INFO] core: unsealed with stored key
2022-07-11T06:20:51.129Z [WARN] core: attempted unseal with stored keys, but vault is already unsealed
These are the only log I see … Also to the service account I have already given the Cloud KMS Admin Permission… is there any other logs that I am missing out on ?
These aren’t complete, I think you left vault running and then ran the init and captured the results of the logs during that time, which isn’t going to tell us anything.
Shutdown vault, delete the logs, delete the pvc and pv … start fresh with just one node and capture those logs. That’ll tell you what is going on with both the storage and init.