Hi,
I’m trying to use Vault for KMS. When I try to create resources using the recovery key process in Terraform I’m getting the following error:
Error: error creating scope: {“kind”:“Internal”, “message”:“scopes.(Service).createInRepo: unable to create scope: iam.(Repository).CreateScope: unable to get oplog wrapper: parameter violation: error #100”}
Boundary logs:
{“id”:“qB0ThPOjby”,“source”:“https://hashicorp.com/boundary/vault1.internal.lan-controller/vault1.internal.lan",“specversion”:“1.0”,“type”:“error”,“data”:{“error”:"recovery KMS was used to authorize a call”,“error_fields”:{},“id”:“e_HrfMGqakOw”,“version”:“v0.1”,“op”:“auth.(verifier).decryptToken”,“request_info”:{“id”:“gtraceid_anFMeg51ZeOugNO5dUTK”,“method”:“POST”,“path”:"/v1/scopes?skip_admin_role_creation=true\u0026skip_default_role_creation=true"},“info”:{“method”:“POST”,“url”:"/v1/scopes"}},“datacontentype”:“application/cloudevents”,“time”:“2021-08-24T09:54:19.426085532+08:00”}
{“id”:“h3J7Hv3hL7”,“source”:“https://hashicorp.com/boundary/vault1.internal.lan-controller/vault1.internal.lan",“specversion”:“1.0”,“type”:“error”,“data”:{“error”:"kms.(RootKeyVersion).Decrypt: error occurred during decrypt, encryption issue: error #301: error unwrapping value: Error making API request.\n\nURL: PUT https://vault1.internal.lan:8200/v1/transit/decrypt/boundary_key\nCode: 500. Errors:\n\n* 1 error occurred:\n\t* cipher: message authentication failed\n\n”,“error_fields”:{“Code”:301,“Msg”:"",“Op”:“kms.(RootKeyVersion).Decrypt”,“Wrapped”:{}},“id”:“e_HrfMGqakOw”,“version”:“v0.1”,“op”:“kms.(RootKeyVersion).Decrypt”,“request_info”:{“id”:“gtraceid_anFMeg51ZeOugNO5dUTK”,“method”:“POST”,“path”:"/v1/scopes?skip_admin_role_creation=true\u0026skip_default_role_creation=true"}},“datacontentype”:“application/cloudevents”,“time”:“2021-08-24T09:54:23.208892093+08:00”}
{“id”:“Jo39WIQkQF”,“source”:“https://hashicorp.com/boundary/vault1.internal.lan-controller/vault1.internal.lan",“specversion”:“1.0”,“type”:“error”,“data”:{“error”:"kms.(Repository).ListRootKeyVersions: error decrypting key num 0: kms.(RootKeyVersion).Decrypt: error occurred during decrypt, encryption issue: error #301: error unwrapping value: Error making API request.\n\nURL: PUT https://vault1.internal.lan:8200/v1/transit/decrypt/boundary_key\nCode: 500. Errors:\n\n* 1 error occurred:\n\t* cipher: message authentication failed\n\n”,“error_fields”:{“Code”:301,“Msg”:“error decrypting key num 0”,“Op”:“kms.(Repository).ListRootKeyVersions”,“Wrapped”:{“Code”:301,“Msg”:"",“Op”:“kms.(RootKeyVersion).Decrypt”,“Wrapped”:{}}},“id”:“e_HrfMGqakOw”,“version”:“v0.1”,“op”:“kms.(Repository).ListRootKeyVersions”,“request_info”:{“id”:“gtraceid_anFMeg51ZeOugNO5dUTK”,“method”:“POST”,“path”:"/v1/scopes?skip_admin_role_creation=true\u0026skip_default_role_creation=true"}},“datacontentype”:“application/cloudevents”,“time”:“2021-08-24T09:54:23.209086677+08:00”}
{“id”:“u7Wpf0J6XZ”,“source”:“https://hashicorp.com/boundary/vault1.internal.lan-controller/vault1.internal.lan",“specversion”:“1.0”,“type”:“error”,“data”:{“error”:"kms.loadRoot: error looking up root key versions for scope global with key ID v1: kms.(Repository).ListRootKeyVersions: error decrypting key num 0: kms.(RootKeyVersion).Decrypt: error occurred during decrypt, encryption issue: error #301: error unwrapping value: Error making API request.\n\nURL: PUT https://vault1.internal.lan:8200/v1/transit/decrypt/boundary_key\nCode: 500. Errors:\n\n* 1 error occurred:\n\t* cipher: message authentication failed\n\n”,“error_fields”:{“Code”:301,“Msg”:“error looking up root key versions for scope global with key ID v1”,“Op”:“kms.loadRoot”,“Wrapped”:{“Code”:301,“Msg”:“error decrypting key num 0”,“Op”:“kms.(Repository).ListRootKeyVersions”,“Wrapped”:{“Code”:301,“Msg”:"",“Op”:“kms.(RootKeyVersion).Decrypt”,“Wrapped”:{}}}},“id”:“e_HrfMGqakOw”,“version”:“v0.1”,“op”:“kms.loadRoot”,“request_info”:{“id”:“gtraceid_anFMeg51ZeOugNO5dUTK”,“method”:“POST”,“path”:"/v1/scopes?skip_admin_role_creation=true\u0026skip_default_role_creation=true"}},“datacontentype”:“application/cloudevents”,“time”:“2021-08-24T09:54:23.20923244+08:00”}
{“id”:“DyysYQ41bN”,“source”:“https://hashicorp.com/boundary/vault1.internal.lan-controller/vault1.internal.lan",“specversion”:“1.0”,“type”:“error”,“data”:{“error”:"kms.GetWrapper: error loading root key for scope global: kms.loadRoot: error looking up root key versions for scope global with key ID v1: kms.(Repository).ListRootKeyVersions: error decrypting key num 0: kms.(RootKeyVersion).Decrypt: error occurred during decrypt, encryption issue: error #301: error unwrapping value: Error making API request.\n\nURL: PUT https://vault1.internal.lan:8200/v1/transit/decrypt/boundary_key\nCode: 500. Errors:\n\n* 1 error occurred:\n\t* cipher: message authentication failed\n\n”,“error_fields”:{“Code”:301,“Msg”:“error loading root key for scope global”,“Op”:“kms.GetWrapper”,“Wrapped”:{“Code”:301,“Msg”:“error looking up root key versions for scope global with key ID v1”,“Op”:“kms.loadRoot”,“Wrapped”:{“Code”:301,“Msg”:“error decrypting key num 0”,“Op”:“kms.(Repository).ListRootKeyVersions”,“Wrapped”:{“Code”:301,“Msg”:"",“Op”:“kms.(RootKeyVersion).Decrypt”,“Wrapped”:{}}}}},“id”:“e_HrfMGqakOw”,“version”:“v0.1”,“op”:“kms.GetWrapper”,“request_info”:{“id”:“gtraceid_anFMeg51ZeOugNO5dUTK”,“method”:“POST”,“path”:"/v1/scopes?skip_admin_role_creation=true\u0026skip_default_role_creation=true"}},“datacontentype”:“application/cloudevents”,“time”:“2021-08-24T09:54:23.209334972+08:00”}
{“id”:“lLTGFEviPQ”,“source”:“https://hashicorp.com/boundary/vault1.internal.lan-controller/vault1.internal.lan",“specversion”:“1.0”,“type”:“error”,“data”:{“error”:"iam.(Repository).CreateScope: unable to get oplog wrapper: parameter violation: error #100”,“error_fields”:{“Code”:100,“Msg”:“unable to get oplog wrapper”,“Op”:“iam.(Repository).CreateScope”,“Wrapped”:null},“id”:“e_HrfMGqakOw”,“version”:“v0.1”,“op”:“iam.(Repository).CreateScope”,“request_info”:{“id”:“gtraceid_anFMeg51ZeOugNO5dUTK”,“method”:“POST”,“path”:"/v1/scopes?skip_admin_role_creation=true\u0026skip_default_role_creation=true"}},“datacontentype”:“application/cloudevents”,“time”:“2021-08-24T09:54:23.209418609+08:00”}
{“id”:“beOkXORGTs”,“source”:“https://hashicorp.com/boundary/vault1.internal.lan-controller/vault1.internal.lan",“specversion”:“1.0”,“type”:“error”,“data”:{“error”:"scopes.(Service).createInRepo: unable to create scope: iam.(Repository).CreateScope: unable to get oplog wrapper: parameter violation: error #100”,“error_fields”:{“Code”:100,“Msg”:“unable to create scope”,“Op”:“scopes.(Service).createInRepo”,“Wrapped”:{“Code”:100,“Msg”:“unable to get oplog wrapper”,“Op”:“iam.(Repository).CreateScope”,“Wrapped”:null}},“id”:“e_HrfMGqakOw”,“version”:“v0.1”,“op”:“scopes.(Service).createInRepo”,“request_info”:{“id”:“gtraceid_anFMeg51ZeOugNO5dUTK”,“method”:“POST”,“path”:"/v1/scopes?skip_admin_role_creation=true\u0026skip_default_role_creation=true"}},“datacontentype”:“application/cloudevents”,“time”:“2021-08-24T09:54:23.209489963+08:00”}
{“id”:“sIZUI80kFF”,“source”:“https://hashicorp.com/boundary/vault1.internal.lan-controller/vault1.internal.lan",“specversion”:“1.0”,“type”:“error”,“data”:{“error”:"scopes.(Service).createInRepo: unable to create scope: iam.(Repository).CreateScope: unable to get oplog wrapper: parameter violation: error #100”,“error_fields”:{“Code”:100,“Msg”:“unable to create scope”,“Op”:“scopes.(Service).createInRepo”,“Wrapped”:{“Code”:100,“Msg”:“unable to get oplog wrapper”,“Op”:“iam.(Repository).CreateScope”,“Wrapped”:null}},“id”:“e_HrfMGqakOw”,“version”:“v0.1”,“op”:“handlers.ErrorHandler”,“request_info”:{“id”:“gtraceid_anFMeg51ZeOugNO5dUTK”,“method”:“POST”,“path”:"/v1/scopes?skip_admin_role_creation=true\u0026skip_default_role_creation=true"},“info”:{“msg”:“internal error returned”}},“datacontentype”:“application/cloudevents”,“time”:“2021-08-24T09:54:23.209586534+08:00”}
{“id”:“IGkBeQ5BS0”,“source”:“https://hashicorp.com/boundary/vault1.internal.lan-controller/vault1.internal.lan",“specversion”:“1.0”,“type”:“observation”,“data”:{“latency-ms”:3790.433411,“request_info”:{“id”:“gtraceid_anFMeg51ZeOugNO5dUTK”,“method”:“POST”,“path”:"/v1/scopes?skip_admin_role_creation=true\u0026skip_default_role_creation=true"},“start”:“2021-08-24T09:54:19.419272596+08:00”,“status”:500,“stop”:“2021-08-24T09:54:23.209706008+08:00”,“version”:“v0.1”},“datacontentype”:“application/cloudevents”,“time”:"2021-08-24T09:54:23.209732046+08:00”}
{“id”:“r4IYCilqWC”,“source”:“https://hashicorp.com/boundary/vault1.internal.lan-controller/vault1.internal.lan",“specversion”:“1.0”,“type”:“system”,“data”:{“version”:“v0.1”,“op”:"controller.(Controller).startRecoveryNonceCleanupTicking”,“data”:{“msg”:“recovery nonce cleanup successful”,“nonces_cleaned”:2}},“datacontentype”:“application/cloudevents”,“time”:“2021-08-24T10:09:15.354713097+08:00”}
{“id”:“GYSRWzd5FJ”,“source”:“https://hashicorp.com/boundary/vault1.internal.lan-controller/vault1.internal.lan",“specversion”:“1.0”,“type”:“system”,“data”:{“version”:“v0.1”,“op”:"controller.(Controller).startRecoveryNonceCleanupTicking”,“data”:{“msg”:“recovery nonce cleanup successful”,“nonces_cleaned”:1}},“datacontentype”:“application/cloudevents”,“time”:“2021-08-24T10:11:15.359211519+08:00”}
{“id”:“ZH7eZ1hYuF”,“source”:“https://hashicorp.com/boundary/vault1.internal.lan-controller/vault1.internal.lan",“specversion”:“1.0”,“type”:“error”,“data”:{“error”:"error decrypting recovery info: Error making API request.\n\nURL: PUT https://vault1.internal.lan:8200/v1/transit/decrypt/boundary_recovery_key\nCode: 403. Errors:\n\n* permission denied”,“error_fields”:{},“id”:“e_o72B69YIH8”,“version”:“v0.1”,“op”:“auth.(verifier).decryptToken”,“request_info”:{“id”:“gtraceid_tNJzpQcWD6m1pfxiZBSW”,“method”:“POST”,“path”:"/v1/scopes?skip_admin_role_creation=true\u0026skip_default_role_creation=true"},“info”:{“msg”:“decrypt recovery token: error parsing and validating recovery token”}},“datacontentype”:“application/cloudevents”,“time”:“2021-08-24T10:24:56.128654024+08:00”}
{“id”:“AW4pi2I7Fx”,“source”:“https://hashicorp.com/boundary/vault1.internal.lan-controller/vault1.internal.lan",“specversion”:“1.0”,“type”:“observation”,“data”:{“latency-ms”:25.243276,“request_info”:{“id”:“gtraceid_tNJzpQcWD6m1pfxiZBSW”,“method”:“POST”,“path”:"/v1/scopes?skip_admin_role_creation=true\u0026skip_default_role_creation=true"},“start”:“2021-08-24T10:24:56.117271729+08:00”,“status”:401,“stop”:“2021-08-24T10:24:56.142515014+08:00”,“version”:“v0.1”},“datacontentype”:“application/cloudevents”,“time”:"2021-08-24T10:24:56.142563887+08:00”}
^[[A{“id”:“SNeCWIKbJz”,“source”:“https://hashicorp.com/boundary/vault1.internal.lan-controller/vault1.internal.lan",“specversion”:“1.0”,“type”:“error”,“data”:{“error”:"error decrypting recovery info: Error making API request.\n\nURL: PUT https://vault1.internal.lan:8200/v1/transit/decrypt/boundary_recovery_key\nCode: 403. Errors:\n\n* permission denied”,“error_fields”:{},“id”:“e_IxQqfy6d5R”,“version”:“v0.1”,“op”:“auth.(verifier).decryptToken”,“request_info”:{“id”:“gtraceid_NEoyovuL1Uo1ektkNxGQ”,“method”:“POST”,“path”:"/v1/scopes?skip_admin_role_creation=true\u0026skip_default_role_creation=true"},“info”:{“msg”:“decrypt recovery token: error parsing and validating recovery token”}},“datacontentype”:“application/cloudevents”,“time”:“2021-08-24T10:25:52.685150913+08:00”}
{“id”:“eSThys6Fnj”,“source”:“https://hashicorp.com/boundary/vault1.internal.lan-controller/vault1.internal.lan",“specversion”:“1.0”,“type”:“observation”,“data”:{“latency-ms”:9.564629,“request_info”:{“id”:“gtraceid_NEoyovuL1Uo1ektkNxGQ”,“method”:“POST”,“path”:"/v1/scopes?skip_admin_role_creation=true\u0026skip_default_role_creation=true"},“start”:“2021-08-24T10:25:52.68148896+08:00”,“status”:401,“stop”:“2021-08-24T10:25:52.691053599+08:00”,“version”:“v0.1”},“datacontentype”:“application/cloudevents”,“time”:"2021-08-24T10:25:52.691106919+08:00”}
{“id”:“ves3iFuJHF”,“source”:“https://hashicorp.com/boundary/vault1.internal.lan-controller/vault1.internal.lan",“specversion”:“1.0”,“type”:“error”,“data”:{“error”:"error decrypting recovery info: Error making API request.\n\nURL: PUT https://vault1.internal.lan:8200/v1/transit/decrypt/boundary_recovery_key\nCode: 403. Errors:\n\n* permission denied”,“error_fields”:{},“id”:“e_hAzpjeZQ2E”,“version”:“v0.1”,“op”:“auth.(verifier).decryptToken”,“request_info”:{“id”:“gtraceid_OYpLIXmoEJyrYC1xP3t7”,“method”:“POST”,“path”:"/v1/scopes?skip_admin_role_creation=true\u0026skip_default_role_creation=true"},“info”:{“msg”:“decrypt recovery token: error parsing and validating recovery token”}},“datacontentype”:“application/cloudevents”,“time”:“2021-08-24T10:26:10.778767731+08:00”}
{“id”:“mNDm4VEIzo”,“source”:“https://hashicorp.com/boundary/vault1.internal.lan-controller/vault1.internal.lan",“specversion”:“1.0”,“type”:“observation”,“data”:{“latency-ms”:6.142936,“request_info”:{“id”:“gtraceid_OYpLIXmoEJyrYC1xP3t7”,“method”:“POST”,“path”:"/v1/scopes?skip_admin_role_creation=true\u0026skip_default_role_creation=true"},“start”:“2021-08-24T10:26:10.775835145+08:00”,“status”:401,“stop”:“2021-08-24T10:26:10.781978091+08:00”,“version”:“v0.1”},“datacontentype”:“application/cloudevents”,“time”:"2021-08-24T10:26:10.782027734+08:00”}
I can see the keys have been created in Vault. Please advise, thanks Kevin.