Multiple creates to same key in kv ends in 502 bad gateway

This is the output from minio kes (the vault client). We could change the logic and implement a lock around create key. Right now we are handling “key already exists”, but when bad gateway is thrown we don’t get that. Is this per design and should we change logic or is there a bug somewhere?

kes-695f65ffc5-8dzvp kes {“time”:“2021-09-01T06:59:16Z”,“request”:{“path”:"/v1/key/generate/e3ec4d92-a1ab-4f5a-92c3-2237d75ab374",“identity”:“640fc3c993c595a29906e7642116ad50c7a47787574fb5fb88117bb4e164650e”},“response”:{“code”:200,“time”:10034740}}
kes-695f65ffc5-8dzvp kes {“time”:“2021-09-01T06:59:16Z”,“request”:{“path”:"/v1/key/create/10825d36-5618-472c-b756-4b97a5dcf4f8",“identity”:“640fc3c993c595a29906e7642116ad50c7a47787574fb5fb88117bb4e164650e”},“response”:{“code”:400,“time”:9267195}}
kes-695f65ffc5-8dzvp kes {“time”:“2021-09-01T06:59:16Z”,“request”:{“path”:"/v1/key/create/10825d36-5618-472c-b756-4b97a5dcf4f8",“identity”:“640fc3c993c595a29906e7642116ad50c7a47787574fb5fb88117bb4e164650e”},“response”:{“code”:200,“time”:9962215}}
kes-695f65ffc5-8dzvp kes {“time”:“2021-09-01T06:59:16Z”,“request”:{“path”:"/v1/key/create/10825d36-5618-472c-b756-4b97a5dcf4f8",“identity”:“640fc3c993c595a29906e7642116ad50c7a47787574fb5fb88117bb4e164650e”},“response”:{“code”:502,“time”:12369894}}
kes-695f65ffc5-8dzvp kes {“message”:“2021/09/01 06:59:16 vault: failed to create ‘kv/10825d36-5618-472c-b756-4b97a5dcf4f8’: Error making API request.\n\nURL: PUT https://vault.vault.svc.cluster.local:8200/v1/kv/10825d36-5618-472c-b756-4b97a5dcf4f8\nCode: 403. Errors:\n\n* 1 error occurred:\n\t* permission denied\n”}
kes-695f65ffc5-8dzvp kes {“message”:“2021/09/01 06:59:16 vault: failed to create ‘kv/10825d36-5618-472c-b756-4b97a5dcf4f8’: Error making API request.\n\nURL: PUT https://vault.vault.svc.cluster.local:8200/v1/kv/10825d36-5618-472c-b756-4b97a5dcf4f8\nCode: 403. Errors:\n\n* 1 error occurred:\n\t* permission denied\n”}
kes-695f65ffc5-8dzvp kes {“message”:“2021/09/01 06:59:16 vault: failed to create ‘kv/10825d36-5618-472c-b756-4b97a5dcf4f8’: Error making API request.\n\nURL: PUT https://vault.vault.svc.cluster.local:8200/v1/kv/10825d36-5618-472c-b756-4b97a5dcf4f8\nCode: 403. Errors:\n\n* 1 error occurred:\n\t* permission denied\n”}
kes-695f65ffc5-8dzvp kes {“time”:“2021-09-01T06:59:16Z”,“request”:{“path”:"/v1/key/create/10825d36-5618-472c-b756-4b97a5dcf4f8",“identity”:“640fc3c993c595a29906e7642116ad50c7a47787574fb5fb88117bb4e164650e”},“response”:{“code”:502,“time”:14019407}}
kes-695f65ffc5-8dzvp kes {“time”:“2021-09-01T06:59:16Z”,“request”:{“path”:"/v1/key/create/10825d36-5618-472c-b756-4b97a5dcf4f8",“identity”:“640fc3c993c595a29906e7642116ad50c7a47787574fb5fb88117bb4e164650e”},“response”:{“code”:502,“time”:14278181}}
kes-695f65ffc5-8dzvp kes {“time”:“2021-09-01T07:02:19Z”,“request”:{“path”:"/v1/key/create/5382d29e-7a9e-4a6f-801a-ba39e046e3f5",“identity”:“640fc3c993c595a29906e7642116ad50c7a47787574fb5fb88117bb4e164650e”},“response”:{“code”:200,“time”:12777254}}
kes-695f65ffc5-8dzvp kes {“time”:“2021-09-01T07:02:19Z”,“request”:{“path”:"/v1/key/generate/5382d29e-7a9e-4a6f-801a-ba39e046e3f5",“identity”:“640fc3c993c595a29906e7642116ad50c7a47787574fb5fb88117bb4e164650e”},“response”:{“code”:200,“time”:9133133}}
kes-695f65ffc5-8dzvp kes {“time”:“2021-09-01T07:02:19Z”,“request”:{“path”:"/v1/key/create/cbd2f428-94ea-42d6-b5d3-71020d749e7e",“identity”:“640fc3c993c595a29906e7642116ad50c7a47787574fb5fb88117bb4e164650e”},“response”:{“code”:200,“time”:11654871}}
kes-695f65ffc5-8dzvp kes {“time”:“2021-09-01T07:02:19Z”,“request”:{“path”:"/v1/key/create/cbd2f428-94ea-42d6-b5d3-71020d749e7e",“identity”:“640fc3c993c595a29906e7642116ad50c7a47787574fb5fb88117bb4e164650e”},“response”:{“code”:502,“time”:16055873}}
kes-695f65ffc5-8dzvp kes {“message”:“2021/09/01 07:02:19 vault: failed to create ‘kv/cbd2f428-94ea-42d6-b5d3-71020d749e7e’: Error making API request.\n\nURL: PUT https://vault.vault.svc.cluster.local:8200/v1/kv/cbd2f428-94ea-42d6-b5d3-71020d749e7e\nCode: 403. Errors:\n\n* 1 error occurred:\n\t* permission denied\n”}
kes-695f65ffc5-8dzvp kes {“time”:“2021-09-01T07:02:19Z”,“request”:{“path”:"/v1/key/create/cbd2f428-94ea-42d6-b5d3-71020d749e7e",“identity”:“640fc3c993c595a29906e7642116ad50c7a47787574fb5fb88117bb4e164650e”},“response”:{“code”:502,“time”:16541677}}
kes-695f65ffc5-8dzvp kes {“message”:“2021/09/01 07:02:19 vault: failed to create ‘kv/cbd2f428-94ea-42d6-b5d3-71020d749e7e’: Error making API request.\n\nURL: PUT https://vault.vault.svc.cluster.local:8200/v1/kv/cbd2f428-94ea-42d6-b5d3-71020d749e7e\nCode: 403. Errors:\n\n* 1 error occurred:\n\t* permission denied\n”}
kes-695f65ffc5-8dzvp kes {“time”:“2021-09-01T07:02:19Z”,“request”:{“path”:"/v1/key/create/cbd2f428-94ea-42d6-b5d3-71020d749e7e",“identity”:“640fc3c993c595a29906e7642116ad50c7a47787574fb5fb88117bb4e164650e”},“response”:{“code”:400,“time”:17492000}}
kes-695f65ffc5-8dzvp kes {“message”:“2021/09/01 07:02:19 vault: failed to create ‘kv/cbd2f428-94ea-42d6-b5d3-71020d749e7e’: Error making API request.\n\nURL: PUT https://vault.vault.svc.cluster.local:8200/v1/kv/cbd2f428-94ea-42d6-b5d3-71020d749e7e\nCode: 403. Errors:\n\n* 1 error occurred:\n\t* permission denied\n”}
kes-695f65ffc5-8dzvp kes {“time”:“2021-09-01T07:02:19Z”,“request”:{“path”:"/v1/key/create/cbd2f428-94ea-42d6-b5d3-71020d749e7e",“identity”:“640fc3c993c595a29906e7642116ad50c7a47787574fb5fb88117bb4e164650e”},“response”:{“code”:502,“time”:18667338}}