Vault getting unresponsive after too many open files error

We are using vault (version 1.6.0) as secret engine for elasticsearch. Vault uses elastic-certificates.crt.pem file to connect elasticsearch. We are periodically reading elasticsearch dynamic credentials from vault.

Usually after few days we starts getting below error and vault becomes unresponsive.

io.vertx.config.vault.client.VaultException: Vault responded with HTTP status: Internal Server Error Response body:{“errors”:[“1 error occurred:\n\t* unable to get client: open /usr/share/elasticsearch/elastic-certificates.crt.pem: too many open files\n\n”]}

It seems like linux has open files limit (ulimit) set to 1024 and vault does not close elastic-certificates.crt.pem file after it accesses it. After the open file limit reached to 1024 we starts getting above mentioned error in logs continuously.

If we run lsof -uvault command we can see more than one thousand ESTABLISHED TCP connections.

Has anyone faced anything similar or have any idea regarding this? What is the recommenced way to deal with this issue?

Any leads would be appreciated.

Thanks in advance.

Hi! It seems that what you’re seeing isn’t unusual. Have a look at this guidance on Linux operating system tuning for Vault:

1 Like

Thank you for pointing us to the right direction. Really appreciated. :grinning: :clap:

1 Like