Max retries exceeded with url: /v1/auth/approle/login

Vault
,
1

Hello!

Every night we run a bunch of ansible playbooks in check diff mode

Every playbook go to our hashicorp vault and take some variables or another things to put in the code

Everything is working good, but sometimes we see errors like

Error was a <class 'ansible.errors.AnsibleError'>, original message: An unhandled exception occurred while templating '{{ lookup('hashi_vault',\n         'secret=game-backend/data/xxx/vault_vars\n          auth_method=approle\n          role_id={{ deployer_role_id }}\n          secret_id={{ deployer_secret_id }}\n          url={{ vault_url }}'\n        )\n}}\n'. Error was a <class 'ansible.errors.AnsibleError'>, original message: An unhandled exception occurred while running the lookup plugin 'hashi_vault'. Error was a <class 'requests.exceptions.ConnectTimeout'>, original message: HTTPSConnectionPool(host='vault.xxx.yyy', port=443): Max retries exceeded with url: /v1/auth/approle/login (Caused by ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x7fbd9a4014f0>, 'Connection to vault.xxx.yyy timed out. (connect timeout=30)')). HTTPSConnectionPool(host='vault.xxx.yyy', port=443): Max retries exceeded with url: /v1/auth/approle/login (Caused by ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x7fbd9a4014f0>, 'Connection to vault.xxx.yyy timed out. (connect timeout=30)'))"}

On our google load balancer ( which stands before vault in internal network) we can’t find any problems, may be we can tweak some settings in vault?

2

hashicorp help me

how to debug this problem?

3

Looks like the error you shared is from Ansible? Have you reviewed the logs on the Vault side when this happens to see what is happening there?

Some other things I would be looking for to figure this out:

  • Get logs from Vault, the error there may be more helpful as to what is going on (maybe auth lockout from some bad connection.
  • Is there any consistency to this, for example its always Monday at 1am but never Tuesday at 1am?
  • What is your Vault configuration (# of nodes, how are nodes managed, what version, what auth method is Ansible using)?
  • Have you tried connecting directly to Vault and bypassing the load balancer when this happens?