Vault KMS: Error creating resources using recovery key

Hi,

I’m trying to use Vault for KMS. When I try to create resources using the recovery key process in Terraform I’m getting the following error:

Error: error creating scope: {“kind”:“Internal”, “message”:“scopes.(Service).createInRepo: unable to create scope: iam.(Repository).CreateScope: unable to get oplog wrapper: parameter violation: error #100”}

Boundary logs:

{“id”:“qB0ThPOjby”,“source”:“https://hashicorp.com/boundary/vault1.internal.lan-controller/vault1.internal.lan",“specversion”:“1.0”,“type”:“error”,“data”:{“error”:"recovery KMS was used to authorize a call”,“error_fields”:{},“id”:“e_HrfMGqakOw”,“version”:“v0.1”,“op”:“auth.(verifier).decryptToken”,“request_info”:{“id”:“gtraceid_anFMeg51ZeOugNO5dUTK”,“method”:“POST”,“path”:"/v1/scopes?skip_admin_role_creation=true\u0026skip_default_role_creation=true"},“info”:{“method”:“POST”,“url”:"/v1/scopes"}},“datacontentype”:“application/cloudevents”,“time”:“2021-08-24T09:54:19.426085532+08:00”}
{“id”:“h3J7Hv3hL7”,“source”:“https://hashicorp.com/boundary/vault1.internal.lan-controller/vault1.internal.lan",“specversion”:“1.0”,“type”:“error”,“data”:{“error”:"kms.(RootKeyVersion).Decrypt: error occurred during decrypt, encryption issue: error #301: error unwrapping value: Error making API request.\n\nURL: PUT https://vault1.internal.lan:8200/v1/transit/decrypt/boundary_key\nCode: 500. Errors:\n\n* 1 error occurred:\n\t* cipher: message authentication failed\n\n”,“error_fields”:{“Code”:301,“Msg”:"",“Op”:“kms.(RootKeyVersion).Decrypt”,“Wrapped”:{}},“id”:“e_HrfMGqakOw”,“version”:“v0.1”,“op”:“kms.(RootKeyVersion).Decrypt”,“request_info”:{“id”:“gtraceid_anFMeg51ZeOugNO5dUTK”,“method”:“POST”,“path”:"/v1/scopes?skip_admin_role_creation=true\u0026skip_default_role_creation=true"}},“datacontentype”:“application/cloudevents”,“time”:“2021-08-24T09:54:23.208892093+08:00”}
{“id”:“Jo39WIQkQF”,“source”:“https://hashicorp.com/boundary/vault1.internal.lan-controller/vault1.internal.lan",“specversion”:“1.0”,“type”:“error”,“data”:{“error”:"kms.(Repository).ListRootKeyVersions: error decrypting key num 0: kms.(RootKeyVersion).Decrypt: error occurred during decrypt, encryption issue: error #301: error unwrapping value: Error making API request.\n\nURL: PUT https://vault1.internal.lan:8200/v1/transit/decrypt/boundary_key\nCode: 500. Errors:\n\n* 1 error occurred:\n\t* cipher: message authentication failed\n\n”,“error_fields”:{“Code”:301,“Msg”:“error decrypting key num 0”,“Op”:“kms.(Repository).ListRootKeyVersions”,“Wrapped”:{“Code”:301,“Msg”:"",“Op”:“kms.(RootKeyVersion).Decrypt”,“Wrapped”:{}}},“id”:“e_HrfMGqakOw”,“version”:“v0.1”,“op”:“kms.(Repository).ListRootKeyVersions”,“request_info”:{“id”:“gtraceid_anFMeg51ZeOugNO5dUTK”,“method”:“POST”,“path”:"/v1/scopes?skip_admin_role_creation=true\u0026skip_default_role_creation=true"}},“datacontentype”:“application/cloudevents”,“time”:“2021-08-24T09:54:23.209086677+08:00”}
{“id”:“u7Wpf0J6XZ”,“source”:“https://hashicorp.com/boundary/vault1.internal.lan-controller/vault1.internal.lan",“specversion”:“1.0”,“type”:“error”,“data”:{“error”:"kms.loadRoot: error looking up root key versions for scope global with key ID v1: kms.(Repository).ListRootKeyVersions: error decrypting key num 0: kms.(RootKeyVersion).Decrypt: error occurred during decrypt, encryption issue: error #301: error unwrapping value: Error making API request.\n\nURL: PUT https://vault1.internal.lan:8200/v1/transit/decrypt/boundary_key\nCode: 500. Errors:\n\n* 1 error occurred:\n\t* cipher: message authentication failed\n\n”,“error_fields”:{“Code”:301,“Msg”:“error looking up root key versions for scope global with key ID v1”,“Op”:“kms.loadRoot”,“Wrapped”:{“Code”:301,“Msg”:“error decrypting key num 0”,“Op”:“kms.(Repository).ListRootKeyVersions”,“Wrapped”:{“Code”:301,“Msg”:"",“Op”:“kms.(RootKeyVersion).Decrypt”,“Wrapped”:{}}}},“id”:“e_HrfMGqakOw”,“version”:“v0.1”,“op”:“kms.loadRoot”,“request_info”:{“id”:“gtraceid_anFMeg51ZeOugNO5dUTK”,“method”:“POST”,“path”:"/v1/scopes?skip_admin_role_creation=true\u0026skip_default_role_creation=true"}},“datacontentype”:“application/cloudevents”,“time”:“2021-08-24T09:54:23.20923244+08:00”}
{“id”:“DyysYQ41bN”,“source”:“https://hashicorp.com/boundary/vault1.internal.lan-controller/vault1.internal.lan",“specversion”:“1.0”,“type”:“error”,“data”:{“error”:"kms.GetWrapper: error loading root key for scope global: kms.loadRoot: error looking up root key versions for scope global with key ID v1: kms.(Repository).ListRootKeyVersions: error decrypting key num 0: kms.(RootKeyVersion).Decrypt: error occurred during decrypt, encryption issue: error #301: error unwrapping value: Error making API request.\n\nURL: PUT https://vault1.internal.lan:8200/v1/transit/decrypt/boundary_key\nCode: 500. Errors:\n\n* 1 error occurred:\n\t* cipher: message authentication failed\n\n”,“error_fields”:{“Code”:301,“Msg”:“error loading root key for scope global”,“Op”:“kms.GetWrapper”,“Wrapped”:{“Code”:301,“Msg”:“error looking up root key versions for scope global with key ID v1”,“Op”:“kms.loadRoot”,“Wrapped”:{“Code”:301,“Msg”:“error decrypting key num 0”,“Op”:“kms.(Repository).ListRootKeyVersions”,“Wrapped”:{“Code”:301,“Msg”:"",“Op”:“kms.(RootKeyVersion).Decrypt”,“Wrapped”:{}}}}},“id”:“e_HrfMGqakOw”,“version”:“v0.1”,“op”:“kms.GetWrapper”,“request_info”:{“id”:“gtraceid_anFMeg51ZeOugNO5dUTK”,“method”:“POST”,“path”:"/v1/scopes?skip_admin_role_creation=true\u0026skip_default_role_creation=true"}},“datacontentype”:“application/cloudevents”,“time”:“2021-08-24T09:54:23.209334972+08:00”}
{“id”:“lLTGFEviPQ”,“source”:“https://hashicorp.com/boundary/vault1.internal.lan-controller/vault1.internal.lan",“specversion”:“1.0”,“type”:“error”,“data”:{“error”:"iam.(Repository).CreateScope: unable to get oplog wrapper: parameter violation: error #100”,“error_fields”:{“Code”:100,“Msg”:“unable to get oplog wrapper”,“Op”:“iam.(Repository).CreateScope”,“Wrapped”:null},“id”:“e_HrfMGqakOw”,“version”:“v0.1”,“op”:“iam.(Repository).CreateScope”,“request_info”:{“id”:“gtraceid_anFMeg51ZeOugNO5dUTK”,“method”:“POST”,“path”:"/v1/scopes?skip_admin_role_creation=true\u0026skip_default_role_creation=true"}},“datacontentype”:“application/cloudevents”,“time”:“2021-08-24T09:54:23.209418609+08:00”}
{“id”:“beOkXORGTs”,“source”:“https://hashicorp.com/boundary/vault1.internal.lan-controller/vault1.internal.lan",“specversion”:“1.0”,“type”:“error”,“data”:{“error”:"scopes.(Service).createInRepo: unable to create scope: iam.(Repository).CreateScope: unable to get oplog wrapper: parameter violation: error #100”,“error_fields”:{“Code”:100,“Msg”:“unable to create scope”,“Op”:“scopes.(Service).createInRepo”,“Wrapped”:{“Code”:100,“Msg”:“unable to get oplog wrapper”,“Op”:“iam.(Repository).CreateScope”,“Wrapped”:null}},“id”:“e_HrfMGqakOw”,“version”:“v0.1”,“op”:“scopes.(Service).createInRepo”,“request_info”:{“id”:“gtraceid_anFMeg51ZeOugNO5dUTK”,“method”:“POST”,“path”:"/v1/scopes?skip_admin_role_creation=true\u0026skip_default_role_creation=true"}},“datacontentype”:“application/cloudevents”,“time”:“2021-08-24T09:54:23.209489963+08:00”}
{“id”:“sIZUI80kFF”,“source”:“https://hashicorp.com/boundary/vault1.internal.lan-controller/vault1.internal.lan",“specversion”:“1.0”,“type”:“error”,“data”:{“error”:"scopes.(Service).createInRepo: unable to create scope: iam.(Repository).CreateScope: unable to get oplog wrapper: parameter violation: error #100”,“error_fields”:{“Code”:100,“Msg”:“unable to create scope”,“Op”:“scopes.(Service).createInRepo”,“Wrapped”:{“Code”:100,“Msg”:“unable to get oplog wrapper”,“Op”:“iam.(Repository).CreateScope”,“Wrapped”:null}},“id”:“e_HrfMGqakOw”,“version”:“v0.1”,“op”:“handlers.ErrorHandler”,“request_info”:{“id”:“gtraceid_anFMeg51ZeOugNO5dUTK”,“method”:“POST”,“path”:"/v1/scopes?skip_admin_role_creation=true\u0026skip_default_role_creation=true"},“info”:{“msg”:“internal error returned”}},“datacontentype”:“application/cloudevents”,“time”:“2021-08-24T09:54:23.209586534+08:00”}
{“id”:“IGkBeQ5BS0”,“source”:“https://hashicorp.com/boundary/vault1.internal.lan-controller/vault1.internal.lan",“specversion”:“1.0”,“type”:“observation”,“data”:{“latency-ms”:3790.433411,“request_info”:{“id”:“gtraceid_anFMeg51ZeOugNO5dUTK”,“method”:“POST”,“path”:"/v1/scopes?skip_admin_role_creation=true\u0026skip_default_role_creation=true"},“start”:“2021-08-24T09:54:19.419272596+08:00”,“status”:500,“stop”:“2021-08-24T09:54:23.209706008+08:00”,“version”:“v0.1”},“datacontentype”:“application/cloudevents”,“time”:"2021-08-24T09:54:23.209732046+08:00”}
{“id”:“r4IYCilqWC”,“source”:“https://hashicorp.com/boundary/vault1.internal.lan-controller/vault1.internal.lan",“specversion”:“1.0”,“type”:“system”,“data”:{“version”:“v0.1”,“op”:"controller.(Controller).startRecoveryNonceCleanupTicking”,“data”:{“msg”:“recovery nonce cleanup successful”,“nonces_cleaned”:2}},“datacontentype”:“application/cloudevents”,“time”:“2021-08-24T10:09:15.354713097+08:00”}
{“id”:“GYSRWzd5FJ”,“source”:“https://hashicorp.com/boundary/vault1.internal.lan-controller/vault1.internal.lan",“specversion”:“1.0”,“type”:“system”,“data”:{“version”:“v0.1”,“op”:"controller.(Controller).startRecoveryNonceCleanupTicking”,“data”:{“msg”:“recovery nonce cleanup successful”,“nonces_cleaned”:1}},“datacontentype”:“application/cloudevents”,“time”:“2021-08-24T10:11:15.359211519+08:00”}
{“id”:“ZH7eZ1hYuF”,“source”:“https://hashicorp.com/boundary/vault1.internal.lan-controller/vault1.internal.lan",“specversion”:“1.0”,“type”:“error”,“data”:{“error”:"error decrypting recovery info: Error making API request.\n\nURL: PUT https://vault1.internal.lan:8200/v1/transit/decrypt/boundary_recovery_key\nCode: 403. Errors:\n\n* permission denied”,“error_fields”:{},“id”:“e_o72B69YIH8”,“version”:“v0.1”,“op”:“auth.(verifier).decryptToken”,“request_info”:{“id”:“gtraceid_tNJzpQcWD6m1pfxiZBSW”,“method”:“POST”,“path”:"/v1/scopes?skip_admin_role_creation=true\u0026skip_default_role_creation=true"},“info”:{“msg”:“decrypt recovery token: error parsing and validating recovery token”}},“datacontentype”:“application/cloudevents”,“time”:“2021-08-24T10:24:56.128654024+08:00”}
{“id”:“AW4pi2I7Fx”,“source”:“https://hashicorp.com/boundary/vault1.internal.lan-controller/vault1.internal.lan",“specversion”:“1.0”,“type”:“observation”,“data”:{“latency-ms”:25.243276,“request_info”:{“id”:“gtraceid_tNJzpQcWD6m1pfxiZBSW”,“method”:“POST”,“path”:"/v1/scopes?skip_admin_role_creation=true\u0026skip_default_role_creation=true"},“start”:“2021-08-24T10:24:56.117271729+08:00”,“status”:401,“stop”:“2021-08-24T10:24:56.142515014+08:00”,“version”:“v0.1”},“datacontentype”:“application/cloudevents”,“time”:"2021-08-24T10:24:56.142563887+08:00”}
^[[A{“id”:“SNeCWIKbJz”,“source”:“https://hashicorp.com/boundary/vault1.internal.lan-controller/vault1.internal.lan",“specversion”:“1.0”,“type”:“error”,“data”:{“error”:"error decrypting recovery info: Error making API request.\n\nURL: PUT https://vault1.internal.lan:8200/v1/transit/decrypt/boundary_recovery_key\nCode: 403. Errors:\n\n* permission denied”,“error_fields”:{},“id”:“e_IxQqfy6d5R”,“version”:“v0.1”,“op”:“auth.(verifier).decryptToken”,“request_info”:{“id”:“gtraceid_NEoyovuL1Uo1ektkNxGQ”,“method”:“POST”,“path”:"/v1/scopes?skip_admin_role_creation=true\u0026skip_default_role_creation=true"},“info”:{“msg”:“decrypt recovery token: error parsing and validating recovery token”}},“datacontentype”:“application/cloudevents”,“time”:“2021-08-24T10:25:52.685150913+08:00”}
{“id”:“eSThys6Fnj”,“source”:“https://hashicorp.com/boundary/vault1.internal.lan-controller/vault1.internal.lan",“specversion”:“1.0”,“type”:“observation”,“data”:{“latency-ms”:9.564629,“request_info”:{“id”:“gtraceid_NEoyovuL1Uo1ektkNxGQ”,“method”:“POST”,“path”:"/v1/scopes?skip_admin_role_creation=true\u0026skip_default_role_creation=true"},“start”:“2021-08-24T10:25:52.68148896+08:00”,“status”:401,“stop”:“2021-08-24T10:25:52.691053599+08:00”,“version”:“v0.1”},“datacontentype”:“application/cloudevents”,“time”:"2021-08-24T10:25:52.691106919+08:00”}
{“id”:“ves3iFuJHF”,“source”:“https://hashicorp.com/boundary/vault1.internal.lan-controller/vault1.internal.lan",“specversion”:“1.0”,“type”:“error”,“data”:{“error”:"error decrypting recovery info: Error making API request.\n\nURL: PUT https://vault1.internal.lan:8200/v1/transit/decrypt/boundary_recovery_key\nCode: 403. Errors:\n\n* permission denied”,“error_fields”:{},“id”:“e_hAzpjeZQ2E”,“version”:“v0.1”,“op”:“auth.(verifier).decryptToken”,“request_info”:{“id”:“gtraceid_OYpLIXmoEJyrYC1xP3t7”,“method”:“POST”,“path”:"/v1/scopes?skip_admin_role_creation=true\u0026skip_default_role_creation=true"},“info”:{“msg”:“decrypt recovery token: error parsing and validating recovery token”}},“datacontentype”:“application/cloudevents”,“time”:“2021-08-24T10:26:10.778767731+08:00”}
{“id”:“mNDm4VEIzo”,“source”:“https://hashicorp.com/boundary/vault1.internal.lan-controller/vault1.internal.lan",“specversion”:“1.0”,“type”:“observation”,“data”:{“latency-ms”:6.142936,“request_info”:{“id”:“gtraceid_OYpLIXmoEJyrYC1xP3t7”,“method”:“POST”,“path”:"/v1/scopes?skip_admin_role_creation=true\u0026skip_default_role_creation=true"},“start”:“2021-08-24T10:26:10.775835145+08:00”,“status”:401,“stop”:“2021-08-24T10:26:10.781978091+08:00”,“version”:“v0.1”},“datacontentype”:“application/cloudevents”,“time”:"2021-08-24T10:26:10.782027734+08:00”}

I can see the keys have been created in Vault. Please advise, thanks Kevin.

First thought is, does the Vault token you gave Boundary have permission to do Transit decrypt operations? If you use that token yourself, are you able to do encrypt/decrypt operations using those Transit keys?

Did you encrypt the Boundary config file using a config Transit key? If so, the same questions apply to that key as well.

Hi,

Yes I used the token and did successfully encrypt/decrypt using the key specified in the recovery kms config.

The root errors are showing that the main root KMS key is failing to decrypt the values in the database:

PUT https://vault1.internal.lan:8200/v1/transit/decrypt/boundary_key\nCode: 500. Errors:\n\n* 1 error occurred:\n\t* cipher: message authentication failed\n\n",

Did something change with respect to the root KMS?

It looks like a problem with my vault setup. I’m using Raft in HA mode with 2 nodes, one of the nodes is complaining about tls:

“transport: Error while dialing remote error: tls: internal error”

[WARN] storage.raft: failed to get previous log: previous-index=94 last-index=93 error=“log not found”

This was it!. I’d rebuilt Vault Transit backend and the Boundary Database looked like it was still referencing the old key id.

Thanks!