Vault write auth/aws/role/ causing vault to hang. (iam)

Vault
1

Current Environment
AWS ec2 instances, Vault v1.4.3 docker container, Consul 1.2.2 docker container

When running

vault write \
  auth/aws/role/test1\
  auth_type=iam \
  policies=example-policy \
  max_ttl=500h \
  bound_iam_principal_arn=arn:aws:iam::xxxxx:role/vault-aws-auth-role-xxxx

It causes vault to hang for 1-2mins, meaning the AWS health check fails and the instance gets terminated.
I’ve tried running the vault write command inside the vault docker container and it gives the same error Error writing data to auth/aws/role/test1: context deadline exceeded.
We have checked logs, but we can’t seem to find anything useful as at the time of the command vault just hangs.

Any help on the above would be appreciated and let me know if you have any questions

2

Is it just with this AWS auth write that is failing?
Consul 1.2.2 is really dated… is there a reason you’re running that old of Consul with a new Vault version?

3

Yeah it seem’s it’s only the AWS auth write that is failing. The same set up was reproduced in a test environment and we get the same error. With regards to Consul, we have just upgraded vault from 0.11.4 to 1.4.2. We have not got around to upgrading Consul yet, did not think the backend would cause issues to be fair and didn’t see any recommendations to use minimum version of Consul when upgrading vault. We can try upgrading consul in the test environment to see if this fixes the issue.

4

So after upgrading Consul from v1.2.2 to v1.7.5 we are still facing the same issue.

5

Yeah I don’t think its consul specific… just was wondering.
Do you have a custom AWS endpoint set?
Might share vault read auth/aws/config/ and see if anyone sees anything odd.
You can connect to the AWS API from the Vault node, right?

6

We’re just using a NLB in front on the vault instances.
vault auth list

Path      Type     Accessor               Description
----      ----     --------               -----------
aws/      aws      auth_aws_0e52e239      n/a
token/    token    auth_token_9732820c    token based credentials

vault read auth/aws/config/

No value found at auth/aws/config

Because we’re using IAM auth method without inferencing, then i believe no credentials are necessary. Assuming this is why there is no value found.

Any yeah Vault nodes can talk to the AWS API.

The strange thing is, there are the odd occasions that the vault write auth/aws/role/test1 works fine.